SANS Riyadh AI & Cloud Security 2026: Integrating AI/ML into SOC Detection Engineering: Building Smarter, Faster Defenses (Video)

--- Page 1 --- you you you okay welcome back you okay welcome back Thank you. Thank you. Okay, for the folks online, we're going to get started in just a couple of minutes. Yeah, I just saw that. He's probably not on this session. Yeah. Okay. Three different things. I'm looking at Slack channel, you know, notifications and all this other stuff. So. Okay. I don't think there's anything for you to take. Should be good. Are you going to do an introduction? No? Okay. Actually, would you ask him, the AV guy, to turn up the levels just a little bit? He might hear that. Yeah, the sound for the back of the room to make it a little bit higher. Because nobody ever sits in the front of room for. Yeah, yeah, yeah, exactly. I mean, for me, I'm either at the very front or at the very back. It's the way that, and for, and a lot of the times, the reason why I like sitting in the back of the room is because I have a tough time sitting down for a really long time. I tend to move around. So I'll like, I'll be doing this, like instead of, instead of sitting in a chair, I'll be like this. And I actually was sitting in this class when Dave Holzer was teaching. I was sitting in this class when Dave Holzer was teaching. Yeah. When I was like ramping up to, to, to teach it myself, at one point he stopped and he was just like, Hey, Mr. Ripa, it was like, he's, he says to me, are you doing yoga back there? I said, yeah. Yeah, I am. I'm doing yoga. He was like, okay. I was like, I can't sit still. It's like, it's hard for me to do that. Yep. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. That's the company name, the Montance. That's, that's mine. I actually, I work for myself, I do the SANS training and then I do consulting and a bunch of stuff. I'm outside of it. I don't do a lot of different consulting. I usually have like three or four different customers at any given time because it's just me. You know, I haven't gone down the path of like, I want to have a shop of people doing stuff. I don't want to have to deal with managing people. I have been a manager in a number of places and it's not something that I want to keep doing I would much rather just work on things that are interesting projects for me we'll get started in just a minute good to see you all right for those of you who don't know me I want to introduce myself just briefly here I'm going to get started there are a bunch of people who are online as well I think that there are about 30 or so people online attending so my name is Christopher Crowley --- Page 2 --- you're welcome to call me Chris I've been a certified SANS instructor for a long time I think it's 13 years and I've been working in computers for a really long time you know my first job in an office doing a lot of work doing computers was in the 1980s and I was doing tape backup stuff for for Unix systems and I've been working in IT and computers since then and a various number of things my last full-time job I was the incident response manager for the office of the chief information officer for the U.S. Department of Energy that was my last full-time job and I left there full-time employment and started doing things like doing contracting work teaching for SANS and I've worked for a number of different organizations in defense U.S. DOD environment I used to travel to the various computer network defense service providers at their combatant commands around the world and I've done work for DOD for years I've worked in educational environments for financial institutions for telecommunications software development development environments. I've done a lot of different things. And a lot of that has been somewhat self-trained. And I just sort of focus on the kind of continual development. And what I do now outside of SANS is I do consulting largely around security operations, designing, building, and operating security operation centers. So that's where I'm coming from for the talk tonight. A machine learning is a thing that has been burgeoning and exploding into this space currently. And a lot of people are trying to figure out, what can I do with this right now? Because so many different things are changing. We're trying to implement stuff. So I am trying to provide some guidance to you tonight on specific things that you can do in your environment for cybersecurity operations. I have been... I've been writing the SANS SOC survey for the last 10 years. I use that as a research effort. And I basically am doing this to be able to kind of keep a pulse on what people are doing in their cybersecurity operations. And for me, when I say security operations, I think of it as the engagement with the organization in order to do business alignment, the monitoring of network and hosts, threat intelligence. Intelligence ingestion, the ability to do incident handling, the ability to maintain a postural assessment of the environment, like things like vulnerability management, and develop exercises to be able to effectively keep people up to date on what they need to be doing and train and assess people in their performance. Because a lot of what we do in cyber is we have some routine stuff that happens, but then we also have to phase shift and think about different things. And so I'm thinking about different things on an ongoing basis. So when I am thinking about the SOC or cybersecurity operations, it's that whole gamut for this. A lot of times people ask for the slides. --- Page 3 --- The slides will be available to download from my website. It's montance.com slash prez, short for presentations. And that actually is a public repository of a lot of talks that I've done. And I've been writing talks and stuff. I've been in cybersecurity for more than a decade. And I've tried to make as much of that public as possible. And I just share a lot of information. And this is the easiest place to get to that resource. There's also a paper that I wrote through Hacking News and AI that has some of the same thoughts that I'm talking about tonight, if you want to reference back to that. This mgt517.com is a link shortener domain. And so the hn-ai will get you all the information that you need to get you to the actual resource that's on there. Okay, so I want to start with some definitions. Just to make sure we're talking about the same, same thing. When I say AI, or when I say ML, I want you to understand what that is. And there are a bunch of different attributes of this. I'll give you sort of the canonical definition of what AI is. And really, artificial intelligence is a large domain of practice. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're talking about today, we're talking about artificial intelligence. And most of the things that we're concept of narrow AI and the concept of narrow AI is a system designed to solve one very specific problem. And you'll hear a lot of talk about general or strong or AGI or artificial general intelligence. And I'm going to comment on that in just a second. But mostly what we do in cybersecurity right now is narrow, narrow AI. That's really all that I'm going to talk about tonight. I will make a brief mention of AGI in just a second. The canonical reference, if you're looking to define what artificial intelligence is, is this Russell Norvig textbook. There's pretty much what everybody agrees is like, this is the book that defines what this is. --- Page 4 --- And so they have a concept of what artificial intelligence is supposed to provide to us. Efficiencies in problem solving, the ability to constrain the operation to to be able to look at things, knowledge representation, natural language processing. And ChatGPT kind of blew the lid off of the natural language processing for a lot of people. It is such an amazing capability to be able to interface with a computer system and just write in your language and communicate in that way. That aspect of it is just one part of what AI is supposed to be able to do. It's supposed to be able to learn based on activity, experimentation, and with definitions of objectives to be able to accomplish whatever those objectives are. Learning is the iterative reintroduction of information in order to be able to adjust itself to get better. It should be able to do observation and activity. And so robotics are actually another area where AI is really making a difference. It's making a huge impact. So since everybody seems to be asking about it or claiming that it's coming, this is actually from a Sherwood News article that was published recently, where they didn't ask all of them, but they took published information for a number of thought leaders, like people who are running the companies that would be the likeliest candidates of producing AGI. And they basically looked at the various things that they said in terms of when it's coming. And this is actually on the Habib's interview. I'm sorry, that's Hasabi's interview, who is part of Google, effectively. And so some people are predicting AGI is already here. I don't agree with that. I think probably you don't agree with that either. The concept of the artificial general intelligence is a thing that has human level understanding and reasoning. We're still in narrow circumstances. And personally, I think it's going to be five, 10 years before we get something that really, truly embraces this concept of AGI. So far out in the space, we don't really need to worry about it. But what we do need to worry about, what we do need to think about. And the answer to that is, what can I do now to help to make everything more efficient and capable in terms of what we're supposed to be doing in the domain of cybersecurity? And so this Russell Norvig book, they talk about the different techniques and uses, things like knowledge representation, strategic planning. These are all things that we currently can do. And I would like you to bear in mind, if you're going to be a cyber security expert, bear in mind that there's a lot of stuff that you can do with AI and ML. But we need humans to be present. I think that people are afraid for their jobs in the cybersecurity space because of AI. And I want to reassure you, your jobs are not going away, not anytime soon. --- Page 5 --- Not because of AI. Not because the cyber problem is going to be solved. There is a tremendous amount of work that is yet to be done, and it's simply not going away. So my encouragement for you is to think about how you leverage these tools to make yourself more capable and get rid of the stuff that you don't need to spend your time doing because it is in some way replaceable. Don't be afraid of taking tasks away from yourself. Be more afraid of not embracing the tool and not understanding how to use it. So machine learning is the iterative tuning of algorithms in order to be able to provide predictions on data that it has not seen before. We have concepts of security. Supervised and unsupervised learning. This is an important distinction. And you hear people talk about, I hear people talk about this in ways that it's pretty clear to me that they don't understand the difference. Supervised learning requires labels to enable the machine learning to occur. And the machine is going to learn how to arrive at the labels that it's told, that, these are the right answers. And what we want with the machine learning is for it to be good at predicting on data that it hasn't seen previously, given a reasonable sample of data of the type. Unsupervised learning has no labels. You'll hear things like self-supervised learning or semi-supervised. The idea of that is that if I have something that has some supervised learning, and then it can take its performance, and assess it, and additional data that is like what it worked on initially, it can then do some unsupervised learning on that data, and then continue to progress its training in order to end up making its supervised learning better. Another thing that you'll hear as a term when we hear these terms around supervised learning is human supervised learning. Human supervised learning is the concept that we look at the output with people in order to verify if the tool is doing what it needs to do properly, and then adjusting labeling to be able to continue the training in the supervised learning approach in order to be able to make the algorithm better. And still, I haven't got into what I want you to be doing. I'm still just giving you vocabulary terms so that when I start talking about this stuff, if I actually mention these terms, you understand where I'm coming from with them, and also because there is a lot of misinformation in this space, and people don't keep the terms straight. So we'll hear about things like training data and validation data or test data. This is used in order to get the model to adjust its internal weights. The training data is used for that. And test is something that it doesn't get to see while it's training. And then we use that to evaluate whether it's working well or not. --- Page 6 --- If you're going to do something that's not going to work, you're going to need some sort of machine learning in your environment. You need a reasonable amount of data in order to do that. It would be great if you had millions or hundreds of millions of labeled samples. But you don't need that much in order to do machine learning. If you have thousands of examples, you might be able to do machine learning for it. I recently did an engagement with a customer in order to do a prototype where they had some very specific, specific internal legal needs. And a consultant that they had engaged with for a long time, or maybe it was an employee, had left. And they asked me, hey, can you develop machine learning to replace this expertise? I said, we have some stuff. So I went and I did a prototype with the data that they had. And my finding on the prototype was, I just don't think there's enough data for me to go on here. You basically are giving me like 40 samples of things for this fairly complex legal decision making. I said, I don't think we need to. You don't need to spend any more money on this. There's just not enough data for me to apply these techniques. So there is something where it's like you just don't have enough data to move forward. But if you have thousands and thousands and thousands of records. of something, that's probably enough to move forward. There's another term that's an important one. We hear about this. It's the concept of overfitting. Overfitting isn't necessarily bad. Overfitting is an indication that the model has arrived at a place where it's begun to just memorize the details of the data that you're feeding to it during training. Actually, Francois Chollet, who's the person who wrote Keras, which is a very important library in machine learning, says, if you're going to be working on a problem, you know that you can use the machine learning to do it if you can get a model to overfit on it. Because then you can adjust what are called the hyperparameters of the model to be able to say, like, OK, let me adjust the complexity of this so that I get it to where it's both able to parse this amount of data that I want it to give to it as well as sort of simple enough to be able to generalize. If you get it to overfit, it has stopped generalizing and just started memorizing the answers. Have you ever spoken to a person who pretty much has memorized a bunch of terms, but they can't really make sense of it in a conversation? They're overfit. They've just memorized the words. Overfitting is like that in a model. We also have the concept of reinforcement learning in this training where it's like kind of pushing the model. Hey, this is what you should learn. --- Page 7 --- This is what you should learn. I'll talk a bit about deep learning neural networks. I'll talk about some other techniques that are available. Deep learning neural networks process information, data that's available. They make substantial use of hidden layers. Hidden layers are basically layers of processing that we don't get to see what's going on inside of them with regard to the math that's being adjusted during the training. The calculations and the hidden layers are what people refer to as the weights and biases. When you hear about weights of a model, what that means is the learned parameters inside of the model. There is a structure to a model for deep learning. There is a structure to a model for learning neural network, which is the hyperparameters of the model. The internal adjusted numbers, which if you want a really simple example of it, it's basically the slope and intercept of a bunch of lines. It's not quite that simple because it's not lines. They're actually tensors, which is a linear algebra concept, which is a highly dimensional mathematical object that allows for the performance of mathematical transformation on data. And what we do in a deep learning neural network is we prepare the data that we wanted to learn from. This can be an image. This can be some text. This can be some network traffic. We prepare the data. We push it into the network. The network processes it by basically just calculating the math and then arrives at an answer at the end. And when you build a deep learning neural network the first time, the first sample that you push through, it just calculates some output. How does it know what numbers to use? It is initialized randomly at first. These weights and biases that are so ultimately important, they're just random numbers that are thrown in there. And we push some data through it. And then, because this is supervised learning, we have an answer that it's supposed to arrive at. The answer was supposed to be 1, because the options are 1 and 0. It came up with a 0, wrong. And it's told, after it's arrived at the calculation, that it's wrong. And it uses that wrong information, which we refer to as the loss, to be able to decide, how do I adjust all of these numbers to make it so that the next time that data comes through, I'll get the right answer, OK? The optimizer function is the thing that is going to adjust these weights and biases within the hidden layers in order to be able to arrive at the right answer next time. That's how deep learning neural networks are trained. They just adjust the, and I'm going to oversimplify it, the slopes and biases of some lines, or the values and the tensors and the neurons in the network, to be able to arrive at the right answer. And if you've structured the thing well, you've given it enough complexity --- Page 8 --- to deal with the data and constrained it from overfitting so that it generalizes well, it'll work pretty well on things that it hasn't seen before. I hate to spoil. I hate to spoil the magic behind all of this stuff that we're talking about. But that's it, folks. That's what it does. That's it. But the thing is, turns out, that stuff is actually really useful. But that's all the magic inside of it. That's all the magic inside of it, is it's adjusting these values in order to do predictions. So if you take that concept and start doing it, you can do some super cool stuff. I don't know if you've ever played a record. The music's playing. And then there's this concept of if you actually play a vinyl record, there's this warm sound, there's some noise, because of the physical interaction between the needle and the groove of the vinyl. Now that noise isn't exactly the same, but it's there. And what we could do is we could actually say, OK, this is what the music is supposed to sound like. This is what the music actually sounds like. I want you to learn how to take the raw input signal and take some other input signal that's kind of dirty, like it's got this noise inside of that, and learn how to get back to the original input. And this technique is the thing that is implemented in machine learning in a concept of an auto encoder. Oops, sorry. I went too fast. In the concept of an auto encoder. So we can think about auto encoders in order to be able to clean sound. Auto encoders are actually especially useful when we're doing training on data inside of cyber, because you can think about that as sort of like a clean version of what we want. And then we can use that to look for things that aren't what we want. So this is an interesting sort of approach to this. What we want out of machine learning is our predictive capability. And what has actually really substantially changed the environment for us was ChatGPT, November 22, captured the sort of like popular conscience around what machine learning could be. So this is based on the concept of attention and training using selective attention. Self-attention and cross-attention enable generative pre-trained transformers to be able to produce output based on input that it's seen previously, but in a way that is sort of like the concept of an auto encoder. Here's the input. Give me that exact input as output. But instead of it being exactly like that, it's like, give me the feel of that music. Give me a thing that's like that, but you're free to just create whatever makes sense. And this self-attention, cross-attention training technique, leveraging concepts like auto encoders are what give us the generative pre-trained transformer. The reason why I'm dragging you through the details of the vocabulary in the background on this stuff --- Page 9 --- is I really want you to understand that it's still just this predictive math that's been given a little bit of creative license to just spit out answers. OK. So we have a bunch of things that are emerging in the space. The idea of dense transformers, the idea of mixture of expert use where we have things trained on very specialized things that the larger system can leverage an expert specific thing. And my favorite example of this is if I'm talking about pythons, OK. SPEAKER 1. OK. SPEAKER 2. I might be talking about snakes, or I might be talking about a programming language. And if I'm asking for help with my sick Python versus help with my Python program, the thing needs to be able to differentiate which expert it should engage in order to give me the right sort of guidance. And you can see that there's a contextual comprehension of what the question is in order to be able to look at that. Another thing that will come out, and you will hear more about this in the next couple of years, is the latent state space model style. This is going to be more important around control systems, where it's going to be giving objectives on overall performance. And then the space state model will be tracking inner details in order to basically keep things arranged within parameters of operation. It might also be interesting for people, who work in the OT space, to be thinking about this terminology because it will help with control systems. This is the sort of machine learning and artificial intelligence that will be focused on control systems. And you will be hearing more about this over the coming couple of years. So, that's the intro. Some vocabulary stuff. Now I'm going to talk about, given that that's the current state, what's the latent state? What could you be doing right now to apply these things in cybersecurity? My backdrop explanation comes from this McKinsey article. I have a little link shortener to get you to this AI3. This is an article from, I think it was 23, 2023, where they talked about the concept of takers, shapers, and makers. You should be at least doing, the taker, right now. If you work for an organization that has funding at the level to be sending you to a SANS class, you should also probably already be doing some of the shaping of the tools, which is the customization of the tools that are available. Taker is you're just using ChatGPT in the free version online. The tools there, you can use it as it's given to you. Shaper is you can do some customization to your security co-pilot from Microsoft to be able to do that. You can be able to adjust the way that it does it for your specific cases. That's what a shaper would be. And makers are, you have the sophistication, you have the data, --- Page 10 --- you have the capability to be able to train on your own data for things that do exactly what you want them to do. Okay. You decide what you need in given circumstances. And so I'm talking across all of these, but this is a nice way to think about the, like, how much effort do I need to do? Just use it as it's designed. Use it with some adjustment that maybe I need to customize, or maybe I just have enough knowledge to tune it to where it's best for us. Or do I need to build it myself? Okay. Another thing as you're doing this, people always ask me about where do I do the work? You might do it in the cloud. You might do it on your own system. I've been experimenting with, you know, small scale systems that to me are like disposable, where essentially, if I go and engage with an organization and they want me to do some machine learning on their data, but I'm not allowed to, you know, train or take their data out or it becomes a problem. Just like, how about I just show up with a laptop and you keep it at the end? Like that's the model that I'm thinking of for a lot of people for prototyping level effort. If you're at the point where you're a developer that's doing, you know, prototyping stuff that you're really going to actually go into an NVIDIA data center. Look at the NVIDIA Spark as an example. It's an ARM processor with a Blackwell GPU coprocessor in order to be able to actually do that work. Okay. So I'm just trying to give you sort of a concept of if you want to do the work for some of this internal stuff, you need a place to do that. You could also have servers with GPUs that are doing this operations for you. Okay. So I've been doing a thing recently. I just want to tell you about it because I think it's an interesting example. It's not cyber specific, but it's, it's a thing that I've set up using almost entirely free resources to do GPT generation. I'm using my Google workspace environment with a Google sheet with app script and I'm using app scripts to go and scrape information from my own website and other repository of information that I have and then I'm using that information to go scrape off of the SERP sort of server. SERP API to be able to identify things that are interesting terms for from people for what they're searching for. And then I dynamically create blogs on the topic that people are asking about based on things that I've written previously with reference to those things. And this is all basically sitting in a Google workspace environment with some like automated stuff going in, like checking, what people are asking about and then writing it for them and I'm using free API keys with cascading cascading sort of attempts at different LLMs and different resources where it just tries this tries this tries this tries this and that with a little bit of smart queuing it just sits there and works. --- Page 11 --- I get a, I get a daily report created 12 blog posts. I mean, it's, it's just like fascinating. And I cobbled it together. From nothing. I'm not using Claude code to build stuff. I'm using Gemini to tell me what JavaScript I should use to implement these features. And then I'm just iteratively enhancing it over time. Out of free resources. I'm building that myself. Okay. So for cybersecurity specific stuff, I think of five different spaces where we should be doing things. Threat hunting, detection, engineering, software development for analysis, automation, orchestration and reporting and communication. I'm going to speak to each of these, but I kind of need to to. Tell you my opinion about threat hunting and detection engineering before I get into how I want you to do that. My concept of threat hunting is you should actually have a program of threat hunting for the people that are interested in this. And you should be able to do that. That's what I'm doing. I'm not going to. I'm not going to. I'm not going to. the people who work in the security operations. And I think everybody hunts. You might even have dedicated hunters. But I think everybody should be given a little bit of time to hunt, because it is a creative action that helps you to say, I think the system is not working right. I want to investigate whether that's true or not. And I have some talks on my presentation page where I talk about that overall program. But that's what threat hunting means to me. Some people, threat hunting means retroactive analysis using new IOCs. That is not threat hunting, as far as I'm concerned. That's just an action that should be routine, that you just automate away. The IOCs come in. We know that that was occurring over the last six months. We can search for it. Maybe I need a person to do that. But still, that's not threat hunting. I know what I'm looking for. In my opinion of what threat hunting is, I don't really know what I'm looking for. So I need some hypothesis-driven inspection to be able to say, is this working right? And my hypothesis starts from the fact, or from the position, that it's not working right, and that's what I'm looking for. Detection engineering, to me, is a complementary action. When I do some threat hunting, where I'm sort of, like, trying some things and testing it out, I'm going to have some ideas of, like, oh, man, I need the system to be detecting this really fast. If I need it to be really fast, that means I have to have specific objectives for detections. I can build hunts using machine learning. I can build detections using machine learning. But there's a very different approach to how I'm going to build each of those. Hunting, fast, quick, effective. Sloppy. Detection engineering, like, actual engineering. I have very specific objectives for a specific task. --- Page 12 --- And then I go and build that elegantly and efficiently and optimize that. I'm not optimizing hunting. If I get to the point where I have a hunt that I want to optimize, I say, put it in the queue for detection engineering, because it's a good idea for detection engineering. So what that looks like to me, in terms of using AI, for this, I have that hunt program defined. And you can go to this. I call this sort of my hunting bootstrap, starting from nothing. And I just walk through. These are the metrics. These are the pre-hunt plans. This is the reporting that comes out. I'm not going to go down that path now. But if you want it, it's there. I can use the GPT to help me with ideas. Hey. Hey. Hey. Chat GPT. We have these sorts of systems. What sort of things could I hunt on? Have it go do research. You could go manually search for the stuff. Have it go do a collection of ideas of potential hunts. That's how you should be using the AI for that. You have some ideas? Ask it. Another thing that you could be doing for hunting is give me code. Do you have a code that you could use for hunting? I have a code, but I'm going to use a code that I've been using for a while. I'm going to use a code called A1. A1 is the code that gives me the ability to do resource creation, to build an ML to look for this specific thing within this sort of data. Here's a sample of the data. Let's talk about first how we're going to do the extract, transform, and load of this in order to sanitize the data and select the appropriate features. After that, we're going to think about the right style of a deep learning neural network to develop. And after that, we're going to do some data generation to just verify if this is a good or not good way of doing things. And after that, we're going to do some data generation to just verify if this is a good or not good way of doing things. to just verify that it's feasible. So here's this. You build me some synthetic data. And while you're at it, build me a generator for more of that data so that I can have as much data as I want. That's a couple of hours with Gemini to get that working, to be able to develop something that's machine learning for a hunt because you have an idea about what you want to look for. I mean, that's right now already available. And if you're not doing that, start. It's just like, you should be doing that right now. All of that's readily available for you. And if it's maybe not you doing that because you don't have the technical acumen to do it, make sure the people who should be doing it are doing that. --- Page 13 --- We can specifically look for IOCs. Give me code to look. Great example of this. There was a thing that came out for Notepad++. I use Notepad++ on all my systems. And eventually, in Notepad++, there was a targeted compromise of resources for that. And ultimately, there were some indicators that came out from that. So I went to Gemini. I said, write me a PowerShell script to pull Windows event logs to be able to look for these specific indicators. And the first time I ran it, it was horrible, completely naive. And I did what I like to do when I'm dealing with this not working. I put on CapsLock. And I yelled as loud as I could yell about how stupid it was. Because it doesn't care, right? It doesn't matter. And then about six versions later, I had something that I was like, oh, that works pretty well. And then I did something to put some data into the Windows event log to make it seem like that had happened. And it detected it. I was like, OK, cool. I don't think I'm compromised. But that's a pretty good quick hunt, an hour of work. Hunting for a thing that's a new emerging risk to me, that's useful. That's a hunt. We can also do some threat actor attribution. We can use internal reasoning models in order to help us with attribution. Attribution to me is useful because it helps me to investigate other hosts. And if I'm, and I told you already, I've worked at a bunch of government agencies. If I'm actually tracking campaigns over time, I can do that. I can do that. I can build the information that I have internally in order to be able to say like, oh, that might have been them or them. Here's my sort of confidence on different actors based on the information that I'm collecting internally to be able to do attribution. We can also think about large scale data hunt type stuff. I will release something next month from the SOC survey. And I basically. I used it to do a bunch of correlation analysis. And it was terrible. It took me a long time. And I had to actually build the mapping quite a bit. However, GPTs are great at decoding things. If I have something that it's a, you know, like some code that I'm not really sure and I don't have a parser, I can use that to decode. They're also pretty good at giving me exhaustive lists of things with a lot of online research. So you can basically. Basically use it for that. I would suggest to you that if you have one GPT do online research and you compare that to another GPT's output, they'll be different. So what you should do is use a bunch of them, whatever you have access to, whether it's the free one or not. And then have a fifth one that takes the four outputs --- Page 14 --- and synthesizes that for you. Give me all the different information in a single thing. If you even run deep research, say, on Gemini, you'll end up with different answers and two different runs of the exact same prompt in two different instances. It's non-deterministic. So do it iteratively and then absorb that. From a detection engineering perspective, what I want is basically, oops. What I want is something that I want to be able to control the learning. I want to control the training. I want to be able to update it. I want to understand the performance for detection engineering. Now, your vendors are doing stuff. And they're saying, yeah, we have ML inside, and we do this, and whatever. But if I really want to be driving alerts on things that I care about in my environment, I really need to be able to do that sort of development. And so there is an example that we do in 595 where we're actually building multiple autoencoders. And this is just a notional picture of an autoencoder. I feed in the data. I train the neural network in order to be able to output the same data. It learns how to take data in and then spit out the exact same thing. But we force this constraint in the middle, so it needs to drop data. If you had to memorize to be able to faithfully reproduce 1,000 lines, 1,000 words in a row, what you would probably do, and this is an oversimplification, you would probably, out of those 1,000 things, you would decide on 10 things as anchor points within that. And then inside of those 100 things, you'd decide on 10 more anchor points. And then you'd learn to memorize the expansion if you could start at any point within the series. That's kind of what it needs to do. So if you learn a series, one series, and I give you F, and you know that, you know. OK. What comes after F? G. And H follows. If you know that, if I give you a different series, because it's not A, B, C, D, E, F, G, H, I, is the thing that you were supposed to reconstruct, but it's some other series, you're going to do a terrible job reconstructing that series. And this autoencoder approach, training the ML on one specific protocol and having lots of experts trained each on one protocol, we can then judge the confidence of each of those in its ability to do the reconstruction out of an autoencoder to have it say, these five things, they're really experts on this stuff. They don't think it's a protocol that they've seen before. You know what that means? It's an anomalous protocol. We haven't seen it before. Does that sound useful and interesting to you from a detection engineering perspective? Like, that's a really nice application of this engineering concept for our detection. And so this is what we would do. --- Page 15 --- We would then build this. We would do testing. And what's nice about this, this is not general purpose. This is your network, specifically, data for it, trained on data that you know is legit, and under your control to adjust it as the detections work, or don't work, in the future. So this requires a lot of ongoing operational support. This is months worth of work. And you're committing to a long-term development effort, an operational effort. It's a non-trivial task to jump into. But if you want to do the detection engineering, this is a pretty effective thing. Then if you were going to deploy something like that, a lot of times, people say, do we need MCP for that kind of deployment? In my opinion, MCP is the wrong deployment for network detection engineering type stuff. It's good for other things. It's good for providing context to GPTs to be able to get output from them. It's terrible at doing machine learning type assessment, where we're actually hooking up our own deep learning neural network to a interface. You should be thinking about the use of APIs if you're building your own internal deep learning neural networks and training them. MCP is just the wrong thing to think about for that. And you can get some very high-speed server instances in order to be able to accomplish that. And NVIDIA Triton is basically their inference server, is designed around this. And it takes models output from multiple different sort of development environments and in different formats in order to be able to provide the inference as a response. You send the data, it sends the answer back. And it is super fast. You can send batches of these things. So this is the sort of thing that I think in my detection engineering that you should be looking at. Anomalies in logs, account behavior, data loss prevention type inspection. You would look for normal flows or authorized flows, collect data off of those, train for that, and look for things which deviate from them. Known executables versus malware. This is another thing that we do in 5.95 is sort of like an inspection of files. And we build our own detector for that. I have a project that I'm working on where the organization has a lot of design documents for their engineering for the environment that they maintain. That's the core of their business. Lots of engineering, very specific engineering stuff. And what we're doing is using that to be able to look at all the stuff over the entire generation of the creation of the environment. And then within that, using that for future requests to say, yeah, that's good or that's bad, but also to look for deviations that matter from the specification by tracking deviations that are also associated with incidents and then looking for configurations or other things that are relevant. So I can look over time from the engineering documents once I actually see that. --- Page 16 --- And then this can provide architectural guidance on what things they should be doing from a security perspective. This is a non-trivial ask, but it's something that if you're in an engineering-type environment, this is the sort of thought that you should have for detection engineering. These are the design specs. Which of those matter for the things that have gone wrong from a cyber perspective? We can also do this on code review. We hear a lot about mythos. And other things that are out there for it. And mythos, there's this blog post from the Mozilla team who got advanced access to mythos. In this blog post, what they said was, you know, we thought it was bad when we got Opus 4.6. And we had 22 important code fixes to implement. And then we got mythos. And we had 270 code fixes to implement based on the things that mythos found. OK. That's a lot of code. My takeaway from this is the next couple of years are going to be especially busy in the vulnerability management space. I'm telling you, like, there is a storm that is not coming. It's here. This is happening. The vulnerability management in your environment needs to be ready, because there is going to be stuff that is just being dropped on us. And if you can't keep up with that, you're going to get compromised, because those flaws are going to be exploitable very quickly. Doesn't matter if they're controlling the release of mythos or not. Once the patches start hitting, the adversaries can do patch diffs to be able to identify what the problems are. Doesn't matter if they have mythos to be able to identify it. OK. I think you should be currently doing software development for your analysis. OK. And so this is a thing where I already mentioned this a bit. Your analysts in cybersecurity should be writing code to custom implement things right now. This isn't the only thing that they should be doing, but they should be empowered to generate Python or PowerShell to help them to do their work on the fly. OK. In terms of, you know, you're not going to be able to do this on your own. You're not going to be able to do this on your own. You're not going to be able to do this on your own. You're not going to be able to do this on your own. You're not going to be able to do this on your own. You're not going to be able to do this on your own. And so if you have a lot of maintenance and operations, if you think about this on the fly stuff as more durable than just kind of like a throwaway task-specific code, you're going to get yourself in trouble. If you start going down the path of like, let's vibe code this, let's vibe code that, if it's going to be used more than a couple of times, --- Page 17 --- you should actually use good software development practices, even if the end result is not good. Even if the engine is actually the GPT. I'll give you another example of that. I'm working on a project. And again, I've got all these little pet projects that I'm playing around with. I have a lot of, you know, sort of like expertise in the mobile space. So I have a project for an app that I've been thinking about for years that I would really like to build. And what I'm doing with this is I'm actually constraining myself to the use of a local, Ollama instance, and using a very complex architecture for my APK, for my Android development, and only the native Ollama. I'm trying to be able to generate prompts that would allow me to build the APK consistently without error. It's not easy at all to get that going, because I have some substantial constraints. Now. I can remove those constraints and just pay money to have more tokens. But that's not the point for me. The point is, can I do it locally? Can I make that work given those constraints? That's a really interesting sort of approach that most organizations aren't going to take. Like, let's require the rigor of good software development process. And it's actually not that much more to do it. It's just that has to be the required requirement if you're going to build something that's more durable than just a script that you're using for this one thing. Cybersecurity automation and orchestration is another thing that I think is absolutely important and you should be doing it. My opinion is I like agentic substantially. I love the idea that I could actually empower an agent to do something. Because for me, I wouldn't empower the agent to take action until I vetted it. And this is my opinion about SOAR before AI got plugged into it. The idea of orchestration automation, that's just what computers do. We've had orchestration automation for decades. And AI can assist us with selecting what to do and actually executing it. But you have to test it. You have to test it before you let that thing start to run. The other part of this is that, and this is not me making up a term, there is actually a defined paradox of automation. And the paradox of automation is that as automation systems become more robust and more complex, the people who are supposed to be the ones who can intervene when there is a problem become less, and less capable to intervene and save the day when there is a problem. It's just the reality of it. Don't fight it. Be aware of it. And plan for the fact that the people who you're saying, you're the human in the loop, I love that, the human in the loop, they don't know how to fix the problem. So how are you going to deal with that? --- Page 18 --- You need to train them consistently in that sort of like fire drill emergency response for those systems. Otherwise, they will not be ready. And the last thing, and probably the most effective thing right now that cybersecurity should be doing with AI is leveraging it to more effectively communicate with our stakeholders. If you are not currently using AI to enhance your system, then you have to start doing it. You're information reporting, you're way beyond. The nice thing is it's easy to catch up. Just start doing it. But this is an example, my prompt. Using the templates, you have to have templates, using these templates, my executive briefing, my systems administrators briefing, my user briefing, my threat intelligence briefing, and my SOC analyst briefing, generate and send reports based on this ticket. That's a trivial thing to do, especially if you have it plugged into your Slack, or Teams, or chat, or whatever you're using for that. OK, time to send an update. This is this. Grab these things. Make sure to include these. And then just it customizes the message to the appropriate people. I don't need to go look up in the asset management system who's, who's the responsible systems administrator? Oh, if I actually need to? That's a different problem of operations that AI isn't going to solve. And this is the thing that is galling to me about operational implementation of SOAR. People think the AI is just fixing it. It's like, no, you are either like an automation-minded environment before, or you weren't. Now, the nice thing is the AI can help you to become more of that. But the AI doesn't just do that for you. You need to think about automation consistently. I'll answer that question in a second. There was a question from online. And I have something, again, just a personal example, but it is very specific on the security operations type stuff. I wrote a book recently, and all of the chapters, and the book have visualizations that I generated using Gemini to write the Python to be able to produce the visualization. And this is something that I could not have done in the amount of time that I did it in, which means that I couldn't have done it, period. Because I just wouldn't have had the patience to do it. And the idea of this code that's generated, it's, intended to empower people in cyber to basically say, this is what I'm doing for you. This is the value that I'm providing for you. I have my idea of how I articulate that. I wrote it down in a book because I think it's useful for people. But the other part of that is you can just do that with Gemini too, or Claude or ChatGPT or whatever you use. So I think that this is a really important aspect. You should be using deterministic code that was created by a GPT to help you to --- Page 19 --- communicate the status of what you and your team are doing right now. Okay. All right. So two questions that I have online, and then I'll open it up to questions in the room. How do we learn and what do we need to learn to be future-proof in terms of having a solid career for the time coming in the near future in SOC security operations? Engineering space. So you still need the first principles of the details of how the detection engineering, the cybersecurity works. You can't just like disassociate yourself with the nature of how computers communicate, how endpoints are working. This is really important fundamental information. And you're adding to it this sort of like hybrid sort of like developer analytics. You're adding this analyst part to the cybersecurity domain. If you can do that, you're going to be employable everywhere for a long, long time. You have some domain expertise. You have some development capability. Even if it's largely from the GPT, you just know how to do it right. And you're able to do analysis. And what I mean by that is like actual data analysis, whether you call it data science or whether you're using something where you can come up with a result, which is defendable, that is what I mean by analysis. If you can do that, you're effectively unstoppable. You're incredibly useful in the cyber domain or any domain where you plug in that expertise to it. And that's what the GPTs are really giving people right now. There's another one here that's capable of people over time who could intervene. What's your experience on reading, understanding AI-generated code? Is it strictly self-explanatory or a hairball of code, the ultimate level of work, but people don't understand exactly how and why? Okay. So Oliver, for your question, the output from the GPT's code, it is what you give to it. So if you say, hey, give me a quick script that does this, it's going to be an unmaintainable hairball garbage in your words. If you say, use these coding principles, make sure that it's not going to be a hairball of code, it's going to be a hairball of thread trader code. It is very sophisticated so that much of the code is going to be crying. Every single one is just some of the code that is going to be portrayed, but if you say, here again, UPCA does here, and everything has a preliminary data definition and a dictionary for everything that you use in order to make it easy for me to modify these reusable values to customize the code to what I want, it is going to do exactly that. If I say, name the methods this way, annotate every method, provide comments with details. the superior version, if you don't ask for it, it won't give it to you. It's as easy as that. And I will also tell you that it loses its place. --- Page 20 --- It changes things you didn't tell it to do. So I consistently, when I'm using the GPTs to develop code, I consistently keep a running copy of the code. And I say, in the prompt, update this code. And I put the code in again every time. I say, update this code, update this code, update this code. And if you're doing this from a software development environment, what you could be doing is saying, here's the method to update. Do not change this method. You cannot change the API specification to this method. You cannot change the output types. Change these things only. That sort of very direct controlling input is what you need in order to avoid the unmaintainable mess. And that's why I included that on the slide. So my question for you is, right now, are you a taker, a shaper, or a maker? And my further question for you is, what do you want to be? Because you can be whatever you want. It's really that easy. OK? You just need to put the time in. And it's actually really exciting to me, personally. Literally, you just need to put the time in and use this wonderful tool that's available to you in order to enhance your capability. OK? So any questions in the room? Awesome. I appreciate it. Oh, you have a question. Please. Yeah, so in terms of parser generation, I think it's a great application for AI, specifically. Like, this is the sort of data that I need. I don't know of anything that is a direct tool for that, specifically. But if you're doing this consistently and all the time, Cloud Code is probably the choice that you would use in order to be able to basically say, like, this is the thing that I want. It depends on the constraints of your environment, the allowable software, and other things. But in general, I would look for Cloud Code to do that. If I couldn't install that, I might look for, you know, like an Ollama-type instance and then choose an appropriate model that's able to do that. And if you have cloud access, you can also, you know, sort of leverage whatever your organization has available. But if you're asking me what one would I use for, like, a desktop kind of thing, Cloud Code is where I would go down for that. Yeah, so what I would suggest for that is you have a defined test set from the data, and you have objectives, and include those in the prompts in terms of the difficulty of getting this to really understand it. You say, here's the task. Here's the data. Here's some test data that simulates that. Here's the objective. Here's the objective outputs. You verify it yourself to make sure that the code does that before you provide the code to me. And changing some of the inputs to provide testing information. And then you verifying afterwards with testing, --- Page 21 --- which is actually automatable. You know what I mean? You could have that take this, take the code, have it dump that in a file, take that, compile it if you need to, you know, however, and then use that for the parsing, and then know what you should be getting out from a defined test set. That's a great way to be like, nope, error. Do it again. Nope, error. Do it again. Nope, error. Do it again. Right? This sort of like iterative loop as a development approach will actually work, especially if there's an error. If you give it an error, it's going to say, oh, I made a stupid mistake. I'm just a dumb AI. Let me fix that error because I tried to do a thing that was a method call in the current version, but it's actually deprecated in that version. It will see the error and fix it if it's in that iterative loop of enhancement. And that's just the way to think about it. And that's what humans would do. Oh, man, why did I make that stupid mistake? Let me fix it. Or maybe you have to go and read the docs to be able to figure out how to fix it. That's still what you would do. Force it to do that for you. Yep. Yeah. Sure. Other questions? Just thinking about it. Yeah. For example, how are we able to use different technologies? For example, for sim, for threat handling, for threat intelligence. One of the challenges we are facing is that the experience or expertise on every tool, it is very costly to maintain the people who have experience of that. How the authentic AI or AI can support for a person who is not very much good in the tool where there are so many good documents available on the websites? Is it AI can support for a normal engineer to do a troubleshooting process? Yeah. So it's a great question. So he's saying, like, we have difficulty keeping our people up to date. How can I leverage AI to help me to do this? Well, there are a couple of things that I would suggest. The first thing is the pain points that get identified, that there's something that people can't do well. You put that into the SOAR tasking to be able to provide some automation around that to help to minimize that work for people. So what I'm thinking here is, like, if you have a thing where people need to write complex queries on your sim to search for a specific thing, you could probably write something that's going to make a call that says, this is the user input guiding how to write the query, produce the query for the user as a starting point with annotations explaining why this query matches this specific case. And then you give that to the analyst and say, use this query. So you could actually set it up --- Page 22 --- where you're making a call before anticipating what the user needs to do, and that's an automation that gets delivered to the analyst with the ticket that they're supposed to be acting on. That's the sort of thing that I would do, is you identify all those pain points, and those become automation rules and requests to help with the knowledge enhancement, or at least give them the best answer that can be given from that tool. And then the smart analyst, over time, will learn from that and say, okay. Or the analyst might just learn to copy and paste. And if all that you're seeing from the things that come back is that they're getting directly copied and pasted, it's time to just automate that and do it directly and do that automation and have them get the answer back. You see what I'm saying? It's like build it from an automation in a way where it puts the human in the loop, but then the ultimate goal is to just be like, okay, just give them the answer, because it's good enough. They're not changing it anymore. And if they are changing it, you could capture that through logging of the SIM queries and the output of the, you know, whatever the MCP that you're using to do this. And then you just are like, okay, well, these are the enhancements for the automation to be able to give better prompting to fix that problem at scale. And you're leveraging that tasking to be able to help enhance your automations for it. The other part of it is you need to anticipate that you need to keep your people up to date in the knowledge that they have. And there are different approaches to that. Some of this is like, okay, you use this tool, you use that tool. I don't really like it. The other thing is like, here's the repo, ask this tool a question, and it will guide you through the process. And it will give you the results. Okay. Another question. Mm-hmm. Mm-hmm. There. I really like that idea as a concept, using the AI to help with hypothesis generation. And actually, when we shift to the concept of incident response, I actually have a talk from years ago that I was doing on SOAR, minus the AI stuff, because I've been thinking about automation for a while. There's a basis that I tend to use for incident handling for the analysis itself, which is something called analysis of competing hypotheses. It's something written by Richard J. Hauer, who is a CIA intelligence analyst, who basically developed a rigorous approach to managing complex situations and having high-quality analytical outputs. Okay. And part of that is the hypothesis management for what's there. So I would go a step beyond that. I don't just want the analyst like querying, give me a hypothesis. I would suggest that in your incident response, you should think about hypothesis management --- Page 23 --- as part of the approach of, if you want to call it playbooks or whatever, but where you're actually pre-creating potential hypotheses given different incident types. For these are the hypotheses that you start from to just jumpstart that. And I don't need an AI to do that, but you could use AI to do that. And what you could do is actually, again, thinking from an automation perspective, from an incident response approach, given that context, if I have some templates for my analysis of competing hypotheses, where it's hypotheses plus data for specific data sources, I could be updating those templates as I go to make it more and more and more effective to be able to jumpstart that analytical effort. Here's the data that we want for a case like this. This is the default stuff. Agentic, go get me this off of the host. Go get me that. Go get me that. Maybe it needs to interface with other systems, but that's the orchestration aspect of the SOAR. And this is something that you should be doing already. Most people aren't. But if you're going to go to the platform where you're saying, OK, now let's use the AI to help me with my hypothesis management, lift it up even more complex than that to a degree, don't make people query ChatGPT off of the data. Build the template off of the incident type, the current suspected incident type, with the hypotheses that you've observed for that previously and the data sources that you already want, and give them that analytical framework for it. OK? And I have several talks on that topic on my presentations page. There's a question online about where can you see the lecture again? I think that the recording of this will be available from the SANS website. I'll verify that, but I'm pretty sure that the recording of this is available. And then this montans.com pres, I'll have the PDF available for the download of it tomorrow. Give me until tomorrow morning, we have to go to dinner now. Any other questions? I'd like to go to sleep, but it's time to go to the instructor dinner. I'm really jellied. All right. Thank you very much for coming. I hope that there was something of value in there for you, or at least gave you some ideas, maybe some things that you could actually do tactically in order to do a little better. Just if you do a little better every day, eventually you're just like, yeah, I'm actually doing some good. Thank you very much. Thank you.