WWHF Hunting by Numbers FINAL PDF (Video)
Presented by: Christopher Crowley
PresentationThe talk "Hunting by Numbers" by Chris Crowley, a security expert with 20 years of industry experience, focuses on providing a framework for a hunting program within a Security Operations Center (SOC). The talk emphasizes the importance of having a good foundation in security operations before attempting to hunt, and introduces the concept of hunting as a research and development effort that helps to fuel other actions within the SOC. It also highlights the need for a programmatic approach to drive hunting and adaptability in the SOC, and the importance of maintaining some rigor in the approach to hunting.
Featured Clips
"It does turn out, it does generally turn out very well"
"I don't know whether it was Chris who turned me on to Korean barbecue, or whether it was David Fletcher, but between one of the two of them, I've never been the same since"
"Meat sweats takes on a whole new meaning after you"
Full Transcript / Content
--- Page 1 ---
So, Chris Crowley has 20 years of industry experience managing and securing networks. His first job in the field was as an Alteryx and a VMS system administrator at the age of 15. Another early starter, right? He's also a senior instructor for the SANS Institute, the course author for socclass.com, and he holds a multitude of cybersecurity industry certifications. This isn't part of the bio, but I'm sure you've seen him at a SANS conference here or there if you've been to one of them. So, that's Chris. Another one. See, everybody starts early, right? A couple of the earlier presenters we're talking about, they got started at age like six, seven, nine, 10, 12. You were professionally employed as a VMS administrator at 15? That's amazing. 15. I actually interviewed for that position when I was 13, and they didn't give it to me because they were like, this kid is just sort of too young for this, and then they called me back the next year, and they were like, hey, you should actually apply, or we're going to list it again. They gave it to some kid who was a senior in high school, and then when he left to go to college, they contacted me back. And basically, they just needed somebody to run tape, physically and literally run tape at the end of the day to do tape back up, and so they're like, we could train a kid to do this. Let's just go get a kid. Fantastic. Well, I was going to add on to the end of the bio, too. Being a fellow with SANS instructor, if you're ever on the road with Chris, take his advice on the food recommendations, because it normally turns out really, really well. At least last time I remember, it did. It does turn out, it does generally turn out very well. I think I was in part of a group with you once in Vegas that went to a Korean barbecue place, and I didn't know that that was the thing, and that was fantastic. So I agree. Oh, Korean barbecue is amazing. Yeah. When you go to Korea, it's cooked over wood, typically. It's wood charcoal, and it's cooked right at your table. Fantastic. It is amazing. I don't know whether it was Chris who turned me on to Korean barbecue, or whether it was David Fletcher, but between one of the two of them, I've never been the same since. And meat sweats takes on a whole new meaning after you . It does. But anyway, it's good to hear. It's good to be on the road and eat some Korean barbecue. I hope that's going to happen again for us soon. The COVID nightmare. 2023, I think, is the projection. Yeah. Something like that. Right. We'll see how it goes. Start some of those smokers now and just keep it going. I suppose we should give a quick shout out before we let Chris roll with this, a quick
--- Page 2 ---
shout out to our sponsors. So we'll do a quick look here. I got to go back and look at the list. Sorry, guys. John's already mentioned the list several times. Of course, we got Black Hills Information Security. We got Active Countermeasures, GuidedCore is helping IT say goodbye to legacy firewall complexity and a low to simpler, more flexible way to secure your critical assets. I almost did that well. Red Canary, thanks as a gold sponsor for detecting and responsing to security events and advanced threats 24 by 7. Silver sponsors, Elastic. They are running. They're running a SIM demo today at 2.30 PM Mountain and Friday at 10 AM Mountain. Everything you love about the free and open Elastic Stack geared toward the security information and event management. If you haven't ever played with Elastic, it is a fun thing to do, especially if you get into the Elk Stack. Elastic Logstash Cabana, a really nice way to stand up some good stuff, have a lot of fun. We have Farsight Security as well. Boy. We have a lot of sponsors here. Farsight offers real-time and historical DNS threat intelligence. Look at www.farsightsecurity.com. I am totally missing having my beer with Paul Vixie this year that I had when I last saw him actually live at the conference and we had a wonderful conversation about, you guessed it, DNS. DNS. Yeah. Forty North. Chris Trunser and crew provides pen testing, red team assessments, and offensive security. Yeah. Forty North. Chris Trunser and crew provides pen testing, red team assessments, and offensive security. You guessed it. DNS. Prinzipão. This is a real creativity training. If you haven't met Chris, he is one of the smartest guys I know and just a lovely, lovely kind individual. Do spend some time talking to Forty North at some point. Great folks. Good people to work with. Very good training. Yeah, very good training. Exactly. And then finally Scythe, Bryson and the crew! Enabling emulation platform that enables Red, Blue and Purple teams to build and emulate real world adversarial campaigns in a matter of seconds. Yeah. Great to see you. Well done. Thanks. of minutes i've been doing a little bit of work with scythe recently and i may have some fun things coming out there down the road there's a little teaser for you so scythe really one of our great silver sponsors i really love the scythe guys and with that i managed to get us to exactly 11 a.m mountain time and i think i even did the time zone conversion correctly we are going to turn it over to chris crowley to talk about what are you talking about chris i didn't see the title hunting by numbers can you uh throw the presenter my way and then i can put the slide deck out oh you mean we actually have to give you presentation control okay we'll try to do that and now we're going to make chris crowley the presenter so that he can present to us
--- Page 3 ---
perfect there we go numbers and things you take it away chris fantastic so hopefully you're seeing that this title slide showing the hunt by numbers this is basically a talk that i decided to give for this it's a basically a program that i have worked with a couple different organizations to develop from a hunting approach i need i have a ton of slides to go through i need to go through a lot of the initial stuff fairly quickly in order to get things working just a little bit of an introduction of myself one thing that i'm doing right now that i would love your participation in everybody who's attending if you get some value out of this talk feel that it's you're like paying it forward by going and taking the sock survey this is something where i will actually release all of the data at the end of the collection of the data and i'm going to release the entire data set publicly except for the attributable things like people's names and emails and stuff like that um so please uh please do that i basically um usually my presentations are fairly guarded and i'm very direct it's going to be a little bit raw today so uh please uh you know if you're sensitive maybe uh you know just give me the benefit of doubt in terms of that i'm some of my opinions are going to come out a little more than they normally would i've done a lot of consulting for a lot of different organizations organizations that you're familiar with i've also worked u.s department of energy u.s department of defense in the past um large global telecommunications companies names of companies that you're familiar with that i'm not allowed to say based on ndas basically what i do for these organizations is typically help them to get better and a lot of what i do is look at their process procedure methodology i try to adjust their architecture i try to compel their people to be better and so what i'm talking about today is an extraction that i have in terms of hunting and this is a little bit of a contrived uh analogy here but i'm doing sort of the painting by numbers and trying to build this up and i'm going to give you a very prescriptive set of actions which i want you to perform for your hunting okay so bear with me while i go through a bunch of introductory stuff because i feel like a lot of times when i just jump into some things people are devoid of the context of where i'm coming from in terms of talking about security operations so i have the soc class that i have tried to consolidate a lot of this into um i have a ton of other talks and there'll be some links in here to other talks on that as well so i will say that i was certain to take yourắc
--- Page 4 ---
be some links in here to other talks where I'm basically discussing some of these fundamental approaches that I'm going to have in my preface here in a lot more detail. So if you want that, the links will be at the end and are peppered throughout the talk. So the value that I see in hunting is that it basically provides adaptability for an organization's security operation. I just had this thing this last week where I was working with this company. I contacted their support and basically they were really nice, but they did not have any skills in order to do something other than what their standard operating procedure told them. And I mean, I felt like they were just reading from a script. And literally at one point, the person said, look, I can't do that. I don't have training to do anything outside of replace parts that are already in your system. And I hope... Hope that your objective for your security operations is to produce adaptability. And I ask you to consider, can you do that? And I would like you to think of hunting as a research and development effort that helps to fuel other actions inside of your SOC. Now, in terms of threat intelligence and the background, I love this particular graphic, and it's extracted from the Defense Science Board from the U.S. Department of Defense, where they talk about the $10 club, the million dollar club, and the billion dollar club, where the $10 club can actually reuse vulnerabilities, the million dollar club can discover vulnerabilities, and the billion dollar club can introduce vulnerabilities into your environment. And a lot of people, when they start thinking about hunting, the first thing that they start thinking about is going after the billion dollar club. I don't think that's the right approach, frankly. I think you should make sure that you have a good fundamental program and you're good at actually detecting the $10 club before you start going after the million dollar club. And then after that, you can consider yourself stable enough with your program to even consider going after the billion dollar club. So, and again, this is all prefatory stuff, where I'm laying this as a foundation before I get to the notion of how specifically I want you to formulate your hunting approach. So when I talk about a SOC, I actually talk about functional capabilities, talk about a steering committee, a command center. Monitoring, threat intelligence, incident handling, forensics, and self-assessment capability. So I'm going to talk about what I mean by each of these here, just briefly. So a steering committee is basically the oversight and the ability to do interaction with the organization. And I see a steering committee having three phases. If you start without a SOC, your steering committee should be formed before you build the SOC and should help to collect requirements and then direct action to build the SOC, which is the second phase, where the steering committee is providing oversight and guidance
--- Page 5 ---
and feedback. And finally, operational performance of the SOC, where the SOC is feeding information to the steering committee. The constituents are comprising that steering committee and basically telling them, here's what we're doing and here's how we're doing it. In the various organizations that I have worked with, I see a couple of different forms of the steering committee. They don't always call it a steering committee, but they tend to have something like this. In some cases, it'll be some formal governance structure with actual voting. In other cases, it'll be a formal governance structure without actual voting. And this oftentimes effectively takes the form of having a couple people who make decisions about what the SOC is going to do, and then all the other participants in the steering committee basically being the implementers of those various actions. But what I see most often is some sort of ad hoc structure where there's no coherent and consistent approach to making decisions for how to do security operations. This is a massive failure. I don't recommend this. I think that you need this as a basis in order to build things out. Another thing that I think that you need to be able to do from a basis of being able to actually approach the problem where I'm going to talk about use cases in engineering inside of your SOC, I want you to think about how you can articulate the impact of problems. And this is a simplistic approach, but it's actually quite effective once you think about how to implement this. I'm basically taking the US CERT guidance on functional information or recoverable impact. And I have suggested that you define for your analysts what low, moderate, and high is in terms of an impact. And then you give them some guidance of what low-type incidents look like and what moderate-type incidents look like and what high-incidence incidents look like. And then you give them some guidance of what low-type incidents look like and what moderate-type incidents look like. And then with your organization, your steering committee, and your constituents, you assign point values for functional, informational, and recoverable values on a per-system basis. And you can start with a really simple set of containers for this. Servers and workstations would be an easy start. And the organization defines that system's value per impact category. And then the organization defines low, moderate, and high. impact in terms of a quantity. And then you're actually able to, on a per-system basis, be able to articulate a quantitative value associated with security incidents. And this range is basically from zero to 300, because each of those green boxes is a one to 10 scale. Being able to articulate this and have this in place for your SOC is really important. And so when I talk about these functional components of a SOC, I also, and I'm not going to talk about this today, but I need to talk about software and hardware
--- Page 6 ---
and structural arrangements and interactions between the SOC and its constituents and processes and effective operations and orchestration and automation. All this stuff is necessary. And you may actually call your team something different. You may not have things split out into these different functional areas that I talk about. You may have people sort of combining different parts. The work that I've done, the things that I've done, the things that I've done, the things that I've read, the experience that I've had, these are the functions that are necessary for security operation. And so you probably have a few people that do all of this stuff, as well as some outsourced capability in order to support your functionality. And so the command center, in my opinion, is basically a capability which maintains situational awareness across all the systems and keeps the SOC on track on the various work that it's doing. Usually in the command center or part, the monitoring capability is some sort of operational support in order to just keep the systems that the SOC needs running. Small organizations often transfer this back to the IT team. I don't think that's the right way to do it. Sometimes you outsource this to a managed service provider. That's fine too. And this may be something where people just chip in to help support the systems, but you need to have a plan for operations. Monitoring is what a lot of people think about for security operations. And the objective for monitoring, monitoring is fast, accurate detection of issues. Threat intelligence is the aggregation of information into our security operations, either from the outside, the consumption of threat intelligence or internal production and tracking of threat intelligence. Incident response is something that we need to be able to do something about problems which we identify. And for me, I have sort of this cynical view of what incident handling looks like. And I put this into this conversation of the three characters of incident handling. The first one is janitorial services. And this is intended to be we clean up messes. And that's part of what incident handling is. And don't deny it. Don't pretend that it's something more glorious than that. A lot of times we're just doing janitorial services type capability. Sometimes we're firefighters. Sometimes we're actually intervening in an ongoing issue and we help to minimize loss. And sometimes some of you may have apex predator type capability in your incident handling, where you're capable of identifying the presence of an adversary in your environment and being able to remove them when you want after you have actually learned things about them is actually a really powerful notion of incident handling that a lot of people aspire to and very few people actually accomplish. So I have, again, lots of links in terms of further write-ups and details of the things that I'm talking about here. So in terms of security operations, we have forensic capability across our ability to
--- Page 7 ---
investigate hosts, pulling information from there, our ability to investigate network information in order to be able to do detailed analysis. Basically, all of our forensic capabilities are focused on the detailed and authoritative analysis given specific constraints and parameters of things which need to be investigated. This includes reversing the data, reversing the data, reversing the data, reversing the data. engineering. And another thing that a lot of people don't build into their SOC when they have the opportunity to actually start off building a SOC is e-discovery. And if you already have e-discovery, I don't advocate that you pull it into your SOC. But if you don't have it already, it's something that you can build while you're building a SOC because it's effectively the same task of investigation, just looking at data instead of looking for an intrusion set. I also think that the capability of self-assessment is absolutely necessary inside of security operations. And so the various factors that I see in terms of self-assessment include configuration management and monitoring, the ability to identify what's normal, have good baselines, track deviations, and be able to help to understand what's going on in the environment. And I think that's a really important part of that. And I think that's a really important part of that. And I think that's a really important part of assert control over the environment is critically important. Managing vulnerabilities is another critically important portion of this. The idea here that I can actually identify that we have misconfigurations or flaws and then compel the organization to change is something which I want driven out of the SOC. It actually helps the security operations be more effective if we can track and change the organization's security posture. Pen testing is something that I'm I think should be a capability of security operations. And if you have a broad separation between your penetration testing and your security operations, they tend to operate in a disconnected fashion. I want that connection to be better established because when I actually am able to demonstrate that I was able to do something to systems, I tend to drive change. And if I say from my vulnerability management program, you need to patch MS-20000, you need to patch MS-20000, you need to patch MS-20000, you need to patch MS-20000, they're like, we don't care. It's not critical, blah, blah, blah. If I say, I was able to print a check from your accounts payable system, the decision makers are going to say, well, what do we need to do to fix that? And I say, apply MS-20000 and change these things, right? So this sort of demonstration compels action. And this is true inside of security operations too. And I love to have somebody with me. I love to have somebody with me. I love to have somebody with me. Pen test type capability inside of the security operations center in order to provide this internal antagonism to drive change. Be like, you seeing me? Are you seeing me? Why aren't you
--- Page 8 ---
seeing me? This is incredibly powerful. And people call this purple team and all this other stuff. But the idea is have somebody that can actually demonstrate impact or demonstrate bypass and stuff will change. And then the last part of this sort of overview of what I want, security operations is the notion of exercise, right? The notion of exercise is really important because most organizations tell me we don't have the time. We don't have the skills. We don't have the resources. We don't have the capability. We don't have the technology, but you know how you get better at stuff. You practice things. And in addition to that, most people tell me that they don't have time to exercise. I'm like, oh no, not true. The way to get better, the way to be more efficient with the limited time that you do have is actually exercise. And we probably need some people to help to build this in order to actually get us to be better. And exercise, if you love it, great. You already know the power of it. But if you don't love it already, then you usually need some sort of external motivator to make sure that you're doing it. Okay. So my little hunting by numbers, building up the notion here is have a good basis in security operations if you want to hunt, right? There's a lot of related information that's necessary to be able to do that. And so I'm going to talk a little bit about that. There's a lot of related information that's necessary in order to do good security operations if you want to be good at hunting. Okay. And so part of this is actually use case engineering. And I see use cases, security operations center use cases as an engineering effort. A lot of people talk about this in different ways, but my methodology for doing use cases is this set of steps. So the idea of identifying the business relevance first with a scenario that the organization is going to use. And then the idea of identifying the business relevance first with a scenario that the organization cares about. After that, decompose the scenario into a storyboard type sequence. And after that, you go find the data that's relevant to the scenario. Most teams start in data and then try to match scenarios to that data. That, in my opinion, in this approach is wrong. You start with the scenario and then you look for the data that's relevant to that. Once you have data that is relevant to the scenario, you actually go look for detection. And then you go look for the data that's relevant to that. And then you go look for detection opportunities. You look for enrichment opportunities, which includes the information that is not part of your system, but it's external to your systems like threat intelligence and other correlation type information. You may also correlate to other data that's
--- Page 9 ---
available inside of your organization. Like here's the system. Let me go to this authentication server and see who's logged on to that system. And so then you actually implement that. You provide monitoring and assessment. And when I say assessment, I'm not talking about simply a metric for this. I'm talking about actually monitoring and assessment. And when I say assessment, I'm not talking about actually building test cases where I can demonstrate confidence in detection on an ongoing basis. And that detection is both technical as well as my analysts being able to actually act upon the things that come through the SIEM. And so I've got hour-long talks, two hour-long talks on this just alone in terms of how to do this effectively. Another thing that I have a lot of stuff about is security orchestration and automation type tools. The SOAR tools are fantastic, but basically what they do is they provide a GUI into scripting. And so another thing that I say about orchestration automation is it's great if you know what you're doing. Once you know what you're doing and what you're capable of, then you'll be able to actually orchestrate and automate things like a champ. But beyond that notion, I also want to emphasize, do not attempt to orchestrate and automate analysis. The analytical component, of the human brain, is, for the time being, superior to the technology that we have. Help analysts to get the data that they need by orchestrating and automating that action. And the last thing that I'll say about orchestration automation is the paradox of automation says, the more that you automate a system, the less capable people are at being able to recover from failures within that automation scenario. Another thing that's an absolutely necessary component of the SIEM is the ability to automate. And I think that's one of the things that's really important. And I think that's one of the things that's really of being able to actually run security operations is the idea of metrics. And Carson Zimmerman and I actually co-wrote a talk that he ended up presenting at first, where we talk about the metrics that I think that you should be reporting. And I have a couple of metrics at the end here on the hunting program that I specifically cite that you should report. I think you should basically track a lot of data, but I think only a small number of things actually end up getting reported out to your steering. And so these sort of notion-based metrics are really important. And I think that's one of the things that's really important. And I think that's one of the things that's really important. And I think that's one of the things that's really important. And I think that's one of the things that's really important. Cycles are what I want you to be thinking about in terms of how hunting fits into the larger picture. And this is some smart art inside of PowerPoint
--- Page 10 ---
to give an idea here. When I tend to draw procedural diagrams, they're way too difficult to actually digest. And some of the stuff that I'll link to later, there are some actual procedural flows that talk about the actions that you need. Preston Pysh, MD, PhD So in my... My next sort of layer that I'm putting into my picture here, it's the idea of building out the capability of all the stuff that you have in order to support hunting. So one thing that I want to say about hunting as I start to move out of the preparatory stuff and more into the hunting-specific stuff, one thing that I want to say here is I think that everybody on the security team should hunt. Not all the time, but I think that this is a two-hour-a-week type assignment for everybody on the security team. And the reason for this is everybody on the security team, or with security responsibility, sees some stuff that they see as a failure, as a shortcoming, something that they want to fix, there's an opportunity, etc. John Ass Bilтории, MD, PhD, worl? Tom Pl �ت for bypass, everybody has a distinctive perspective and I want to empower them to be able to not only develop themselves by doing some novel type actions, but I also want to empower them to share their ideas back to the team. Now, this also does not exclude having a dedicated hunt team. I can still have people hunting individually as well as having a team that all that they do is hunting, okay? And hunting in this notion is basically a variable type action. Things change, right? And so I'm using a variable here, variable N, to have some variability in my hunting, right? And the idea is that I want to make sure that everybody has the opportunity to actually bring their wildest and craziest ideas into this hunting program. But in order to actually demonstrate the value back to the organization, I need to maintain some rigor in the approach so that I can show, hey, this isn't us just going and screwing around all day. This is us doing something in a creative and novel fashion in order to provide specific value because most of our systems tend to be very rigorous and inflexible, right? Like my help desk that I was talking to earlier. Even if I'm not doing that, I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. I'm not doing that. But the attackers that we're looking for have ultimate flexibility. So we are going to balance our time by actually demonstrating the things that we derive from our hunts and being able to report that. And so this little bit of rigor in the program that I'm going to put around
--- Page 11 ---
the hunting actually lets us be even more creative and more effective and more variable in our approaches. So things, and I'm going to go come back to this, But things that I want, posture enhancements coming out to the organization, wins. And I'll talk about what those are, but these are actions effectively that have been compelled by the hunt for someone, us, the SOC team, or them, the IT team, or them, the business to change in order to enhance security posture. Another thing that I want is a picture of our tools and what they do for us. And this, some of this is us taking our own medicine and making sure that we have a very clear picture and everybody on our team is in lockstep of what our capabilities are in terms of the technology that we're using in order to actually be able to be adaptive when something novel comes up. And this also then counts for practice. And what we're doing is we're building out our adaptability portfolio. And we're doing it in a way that simultaneously accomplishes potentially detections of things that have already gone wrong, as well as us being able to find and mature in terms of our view of what we're attempting to defend. So here's a couple of examples of hunts that I'm going to talk about before I get into my prescriptive steps of what I want you to do. But overall, the pitch that I'm making here for what's going to happen is, what hunting is, is investigation of data available in the environment, presuming that alerting has failed. And so examples of this would be looking for DNS for oddities or TLS certificates for oddities or IP addresses and looking for least frequency of occurrence authentication on given hosts. And there are tools that will help you to do this. But this is the idea of the hunt, right? We're looking for things that will not be covered by, by our existing alerts or are verifying our existing alerts to make sure that we haven't missed something. And so a great place to start for ideas is threathunting.net. If you're not familiar with this, threathunting.net maintained by David J. Bianco, a bunch of information coming out of Squirrel from before Amazon bought them. And there is this one zip file that has about 30 different hunts that tells you things that you could potentially do. And you can get it from GitHub repository. You can get it from GitHub repository. You can get it from GitHub repository. You can get it from GitHub repository. You can get it from GitHub repository. You can get it from that particular website. So lots of good ideas of stuff to actually do for your, for your threat hunting. And what I want coming out of my threat hunting are actual detects, right? I want actual detections. That's my number one, most important priority of being able to actually deliver on this.
--- Page 12 ---
But I also want to count as a win operational capture of the thing that I did in order to make it easier going forward. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that's what I want to do. And that operational capture may take the form of a script. It may take the form of, you know, some other sort of rule that I put in terms of my network IDS. Another thing that I want as an outcome is posture improvement suggestions for the organization of Hey, we're looking here, we noticed this, we should probably change this. Maybe we're not going to change it today. But we need to put this into the roadmap in terms of how we're actually adjusting our environment, because we need to adjust based on what this hunt showed us. We also potentially have ideas for some use cases. And I do not want hunt burdened with having to build the use cases because I think that use cases are an engineering effort. And I've already talked about that as a supporting functionality. Okay. Another thing that we can do with threat hunting is basically sweep our enterprise to look for something that we think might be present that we didn't have alerting coverage for previously. Okay. And you might do this with Yara. You might do this with any other of your tools that you have available. The point of this is that I can either sweep the enterprise for scope of a current suspected incident, or I can perform historical retroactive analysis on data that's available now that I've learned something from threat intelligence that I can apply into my environment. And you may have to do hunts very quickly based on updated threat intelligence in order to actually do this. And so one of the outcomes, and value provided by a threat hunt is the idea of building out use cases. And so what I'm advocating for is come up with ideas during your threat hunt in order to actually then go and build out the use case further. Not all hunts are actually good candidates for use cases. Okay. And some hunts are things that you can just do through a script that you'll actually go ahead and run that repeatedly going forward. Okay. So part of the hunting can be, hey, here's an idea for automation. And then as I do this multiple times, I build this scripting capability out. So eventually what I end up with is just me having a report on the action that I would typically perform or have been performing manually. I send me a report that's the output. And after I do three or four weeks,
--- Page 13 ---
of hunting, I should be in a position where I can do that. Unless there's something that is specifically challenging with that, most of this effort, if you apply it in a programmatic and diligent fashion, and part of the outcome that's defined for your team includes automation as a thing which counts as a win for credit for being able to demonstrate value, then this just reinforces that notion further. And of course, there will be some things which you may decide this specific scenario actually warrants the effort of engineering in order to actually build out a use case. But do not constrain the work of threat hunting with building those use cases. This should be a separate process. This should be a separate effort. Okay. And if you don't have a good program, for building use cases in your sim or wherever you're going to implement them, that's another problem that you need to take care of as well in parallel to building your hunts. I want the hunts to remain lightweight, somewhat experimental. Part of the cycle includes automation for it. And once you've automated it enough, then you move on to some additional variability in order to make it better for the organization and look for potentially new ones. So as you can see, the way that we've implemented threat hunting is very similar to what we've done in threat hunting. It's a really interesting approach that we've taken. And it's not just a way to make things easier, it's a way to make things easier for the organization, the organization itself, and then the organization itself. It's a really interesting approach. So thanks for watching. I'll see you next time. Bye bye. Bye bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. other sources of potentially interesting things that you can do, right? So Eric Conrad has the Deep Blue CLI that looks through event logs looking for interesting things, right? I love the talk that David J. Bianco gave several years ago at the SANS Threat Hunting and Incident Response Summit, because what he did was he actually discussed his general principles for outlier detection. And he talks about mathematical approaches to identifying outliers, which you would then go and investigate. This includes least frequency of occurrence, scatter plots, box plots, and isolation forests. And isolation forests are the mathiest math thing that he talked about on that particular day, where basically, instead of thinking like, oh, I should go look at DNS for these sorts of things, you basically just use an algorithm to identify, here's an outlier, go in and investigate it. You don't necessarily... I don't necessarily even know why that algorithm identified that. And so he has an example of a tool that you can feed data to and some suggestions on how to approach this. If you have somebody that's like a super data geek, have them focus on some of these different outlier detections
--- Page 14 ---
mathematically to deal with your large datasets. That is also part of the hunting that I would want them to be doing. If you have somebody who has a really good basis in experience with incident handling, they probably have something like this. You can download this particular file. This was something that was shared with me privately in terms of what this person called take another look. And so James, who shared this with me, basically gave me this take another look listing. And he says, all right, here's all the stuff that in my experience, if I see it, I investigate further. So you can have the math totally abstracted away from experience, or you can have guidance through experience and details of things that you should actually look into. And so you can download this full list from James Butterworth in terms of this from that mgt517.com slash SOC location. But a couple of examples that he has in terms of... James Butterworth. James Butterworth. James Butterworth. James Butterworth. In terms of like a query that might be performed, or someone's actually doing some interrogation of the system in order to identify the environment, looking for directories that maybe don't make a lot of sense, executables running out of a temp directory. And how would you be able to understand this? Well, based on his experience, he has over time collected this list of this stuff just looks weird. Okay. And in order to have a lot of data, to hunt through, you can, whatever tools you have, you can look through the data. But one, if you're looking at tool suggestions, I would suggest Kansa, which is able to collect and analyze. And you can actually look at the various analytical capabilities that are actually in place. Here's another thing that's an interesting idea for material that you might hunt through. And Jake basically talked about the... The supply chain threats to your organization. And he released Screeth, which is a guide for threat hunting within your supply chain in order to consider what sort of issues you might potentially have and quantify the risk associated with them. Okay. So with that background of information in hand, I'm going to now tell you... What I want you to do for hunting. And this is my plan. When I take... And when I'm working with a customer, I tell them, here's what I want you to do. And they usually say, well, like this thing or that thing or that thing. And I tend to be a little bit sort of adamant at the start because I say, look, all of this plan is actually specifically designed in order to be such a cycled over multiple times, do it my way first and then adapt it to what suits you after you learn from trying my method. Okay. I actually, a quick story and an aside. I used to work in this microscopy suite when I was an undergraduate. I actually did a lot of molecular biology work
--- Page 15 ---
and I did scanning electron, transmission electron, fluorescent confocal microscopes in this microscopy suite that I managed. And there was this wonderful retired microscopist. His name was Lawrence Yatsu, Japanese guy. When I met him, he was probably 75 years old. He had retired and then got bored and went back to work. And so he would just show up at the lab and would do work. And this man was amazing. I remember having a conversation with him one time because I was really upset about something. And he's like, hey, we can talk about politics and we can talk about all this other stuff, but you know what I think would actually be the best thing for you, for us to talk about is methodology. Let me show you how to build a Perlodean slot grid in order to be able to have specimens to be able to actually work on. And he's like, this method is something that not a lot of people can do. I'll show you how to do it and you will basically be able to use this for the rest of your career. Now, I didn't go into microscopy as a career, but it was something that the meta information from what he shared with me was astounding in terms of the value of sharing methodology among people. And so I basically figured out, or I'm sorry, he explained to me how to do it. I figured out a couple of ways that work. I figured out a couple of ways that work. So here's my method. All hunts are going to start with a hunt plan. And I'll give you the data structure for what this hunt plan is. I will also give a data structure for a hunt report. Now, yes, this is super lame. And yes, I'm even a little bit embarrassed about this, but I'm using spreadsheets in order to do this. I've done a bunch of sort of like data normalization sort of work for this. I haven't finished that. Just... Because it's one of 10 medium priority things that I'm working on. The way that I envision this is that these actions would actually link back into whatever ticketing tracking system that you're actually using. But the spreadsheet themselves will actually give you tremendous structure to start with. And then you can throw that away and adapt it to whatever you need, or if you want to write it into a script or whatever. Eventually, I will actually release this as a... As a script that helps to guide people in doing their hunting. And it's something that I started working on and haven't finished just because life is involved and busy and it's not something that I was able to finish in time. So sorry. So these are the things that I see in the data that is actually collected for my plan. This is my hunt plan. Before I start hunting, I depict what I'm
--- Page 16 ---
going to do. And yes, this is a little bit rigorous and it's a little bit annoying, but if you do this to start off with, then it makes it easier for us to be able to share information among the team on who's doing what and what actually was done. So I want a serial number, and then I want an instance number because I can run the same hunt multiple times. I have a name for it. I have some purpose. And here, the purpose is identify the SOC team's tool inventory. The data that I'm going to use, and some of this have adapted from the stuff at threathunting.net. Considerations for collection, the maintenance, the maintenance, the methods that I will use for analysis. And I can change this. I don't have to stick with the methods, but I want to propose what I'm probably going to do. And then if I change afterwards, I can actually adjust that for the next time. And then any references or notes or other things that I need to include in terms of this plan. And so I have this spreadsheet. You can download it. This is basically the structure of the data of how you will collect this and start off. And then every single hunt, I'm going to be able to collect this and start off. And then every single hunt, we'll actually have a report. And each of the reports will basically include this data of what we're actually finding when we do this. Did we get any detects? And by the way, if you do get a detect, you should interrupt the hunt and start dealing with your incident handling. Was I able to capture what I did in some sort of a note file that outlines this so other people can do it more effectively? Just put the link to the file somewhere in there. Is this some sort of a posture improvement that I found that I want to tell back to the organization that we need to do this? And I would expect that you have some form in order to communicate that. And I call that a posture improvement report. It might be a ticket. I don't care what the form is, but whatever that is gets noted in the report. If I created a new rule, a YARA rule, a network IDS rule, anything like that, I actually put the serial number of the one that I created because I'm tracking that. And then finally, just mark this, yes, no. Is this, the candidate for use case so that other people can go and look in your reporting to be able to see like, hey, we may want to build some use cases. What are some of the ideas that are actually there? So this then starts to become a good sort of integrated view at the data that we have. And as I mentioned, the outcomes that I want, actual detects, operational capture and scripts,
--- Page 17 ---
rules, the repeated tasks for us to be more efficient, the ability for us to pre-collect or digest data so that when we need to go look for things later, we can go and collect that relatively easily, new ideas for use cases and posture improvement for the organization. And so at the end of the hunt, the individual hunt reports, because I want each individual that does the hunt to actually do their own report. And then I would aggregate this back into a basically report across all of the data that we have. And then I would aggregate this back of the various actions that were performed for that hunt set. This then allows me to build out metrics. And the metrics that I'm going to report out to the organization involve the detects that we found, the posture improvements that were suggested. If we then later find that the posture improvement was actually implemented through our hunting, we can count that. Systems that were correlated to owners, and then an estimated coverage of known systems by hunting. So how many of the systems that I think that I have were touched by my hunting? And so the idea is that we're showing as a percentage, how broad our reach is in our hunting. And at first, it's going to be pretty shallow in terms of coverage. Expect that. That's okay. Just demonstrate that you're actually improving this. And so these are the specific hunts that I want you to do. All right. The specific hunts that I want you to do. Is first off, hunt your own tools. And so I want each individual within the team to know all the tools that are available for us. And yeah, there may be overlap in this effort, but I want everybody to actually perform this hunt. And a posture improvement that may come out of this is, I actually don't know how to use that tool. I should go and review that. And this sort of drives the culture of improvement. We love to tell other people and other teams how to fix their stuff, but we oftentimes don't do that to ourselves. And if you cannot show me that you're able to hunt your own tools, I have very little confidence in your ability to hunt anything at all in the organization and give me a good feeling about your capability for hunting. Start with your own stuff. Next one that I would do in terms of a hunting program is the idea of some form of least frequency of occurrence. I don't care what least frequency of occurrence action you actually choose to investigate. Pick one. Here's some ideas. There are lots of them out there and do it. Some people call this least frequency of occurrence stuff, long tail analysis. Some people call it stack ranking. Fine. This should be fun though. This should be something cool. You get to decide what you're going to focus on in terms of looking
--- Page 18 ---
for things. But part of this is the rigor of doing least frequency of occurrence as an analytical methodology. In terms of the least frequency of occurrence, I'm going to call it stack ranking. The next one, this is sort of like more medicine. We're going to hunt our constituent systems, right? Whatever you have available in your environment, plus some open source intelligence tools, go find the technology that your organization uses, as well as identify the people who are responsible for this. And part of this as an output should be your own inventory and catalog of systems that are actually defended. And a lot of people in SOX complain that they don't actually know what they're doing. They don't know what they're doing. They don't know what they're doing. They don't actually know what systems they're actually defending. And they are not aware of the defensive topography of what they're trying to protect. Well, fix it. This is how you hunt for this. This is the approach that I think is really, really important. And we can then further demonstrate value of, hey, we complain about this all the time, but guess what? We're hunting it and we're finding this stuff. And y'all can tell us about it, or it can show up in our hunting report that you didn't tell us about it, because we're after this stuff. Another thing that we do is outlier detection. Here, we start with command line length. And the reason why I focus on this one specifically is it is a bridge to communicate with my systems and IT administrators on how they are running their work. We're specifically interested in what attackers are doing. We want to have detection opportunities and data in order to differentiate what our systems administrators are doing from what adversaries are doing. And if they could become more efficient, we can do that. And that's what we're doing. And more operational in terms of having scripts or even better signed scripts with digital signatures, I could actually very easily start having differentiation, which drives use cases for execution of action that is not sanctioned as part of what my systems administrators are doing. So I'm sort of anticipating an outcome that I want in a posture improvement and hunting for something that helps me to find the data to support that. Preston Pyshko It's a little bit shady, but it's the way that I do stuff sometimes. Another thing that I say in terms of our next hunt is the team hunt or hunting party. And so the idea here is that I have a theme. I have something that I'm going to propose. I basically have somebody who's responsible to be able to do this. We start looking through the additional resources. If it were up to me, I'd basically bring a bottle of mezcal to the party. Now, I don't know if you're allowed to do this at work, but I am. So that's all good.
--- Page 19 ---
So if you're allowed to, great, open up your bottle of mezcal, and we're all going to start hunting in an effective way. If you follow these simple steps of what I'm talking about, you can actually start to move through this and get your hunting on and make things a lot better in your environment. Preston Pyshko And being able to meld this metric-driven capability of showing value while at the same time letting us be creative and interesting in our approach helps to thwart some of the frustrations that we have in the environment. It also is effective at driving maturity. And most importantly, it's probably going to find intrusions that you have to deal with. Preston Pyshko And I think that's a really important thing to keep in mind. I think it's going to increase the adaptability that you have in your organization in terms of providing security operations in a way which is not inflexible but nice, but is capable and adaptable. And so as you do this, if you follow that cycle of those first five hunts that I want you to do, over time, you realize, I don't need to keep redoing these same things. I can get better. I can get better. I can get better. I can get better. I can change stuff. And that hunting party is hunting for the next best, greatest idea in order to make things a lot better. And so then you start repeating this with variations. You have new people come in, you give them the easy ones. Let them do the stuff that you've already basically blazed the trail in terms of doing that hunting work. And this is not the majority of people's time. This is a little bit of time each week in order to actually improve your visibility and adaptability. Preston Pyshko So if you want to have some further information, you can download this talk and a whole bunch of other ones that I've done at this mgt517.com slash SOC. And there are a lot of other links that I have out there in terms of things. Now, and I mentioned this earlier, if you feel like this was valuable to you in any way, please do me the favor of taking the SOC survey. That data is incredibly valuable to me and your peers. Preston Pyshko I have a lot of information. I have a lot of information. I have a lot of information. of people who have talked to me about the last three years of surveys where they have used that in order to drive change in their organization because it is a good, authoritative, data-based inspection of the state of security operations. So with that, my conclusion is that hunting provides adaptability to the SOC. And I'd like you to think about whether your SOC is really adaptive and a responsive engine of the SOC. And I'd like you to think about whether your SOC is a preservation of your IT capability and is able to actually advance the security posture of your
--- Page 20 ---
organization. And I suggest that you need a programmatic approach in order to drive hunting. But further, I think that you need to make sure that you separate the engineering from the development efforts and be able to actually show to your organization why you need to do that. Preston Pyshko So I know I'm a little bit over in time. And I've totally been ignoring the Discord channel because I can't both read that and talk about stuff. It's pretty hard to do that. Do you guys have any questions for me? I don't know. BB, we got any questions that popped up there that we want to present? We had another person who started out in molecular biology and is, now, a threat hunter. So apparently, it's the way to go. You don't need computer science-y stuff. You want molecular biology. There's this idea of scientific method. I don't think that we do scientific method in hunting. But I do think that having an analytical methodology and good rigor in your approach is very beneficial for this sort of work. This does feel like a place where that's more true than other places. Lots of places can benefit from that. But if you've got a whole environment and you're looking for... There are things that may or may not be there, and that can be an endless, useless effort if you don't have some focus and some structure and a real methodology about it. So you talk about crossovers, different industries, different backgrounds bring different things to the table. That's a fantastic one for this one. I think a strong science background really does give you a structure to do threat hunting. It really does. It really does. Good communication skills are another thing that's actually really quite beneficial. And Malice, it's Crowley. If you like. I actually, I keep that variable too. Sorry. I'm just looking at the Discord and stuff. Yeah, I was just going to say, which means I probably mispronounced it. Sorry about that, Chris. It's any number of things. It's Crowley or Crowley, whichever you want. I'm fine with that. I know who you're talking about. It probably answers the hey you as well. But anyway. Great talk, Chris. A lot of other names too. Yeah. I've been told the devil on occasion. One of the things I love is hearing about how different people got into InfoSec. And there's some people with some amazingly diverse backgrounds that ended up in InfoSec, which is I'm not quite sure what that says other than that we lead interesting lives and this is an interesting field. I think that's the reflection that I get out of that. And I wonder how much of us are like, you know, we're old and curmudgeon-y because there were no paths to get to computer security. Because for a lot of us, it wasn't really a thing. Like it was just something we kind of stumbled into. And I wonder if that's going to continue to be the case. Or do you think
--- Page 21 ---
that the people with the well-defined paths are going to be the ones that are going to kind of take over? Well, my opinion is it's going to actually, there's going to be a small pocket of people who are doing this research and development type stuff. And then a lot of other people are going to be doing this sort of more routine engineering type stuff. Yeah. Right. So you don't think those are the people who are going into cybersecurity operations programs. I'm sorry, John, what did you say? I was going to say, so you don't think it's going to be like the jack of all trades that it has been where I'm running, working in a sock all day long, I'm doing system administration and I'm doing buffer overflows or heap overflows or rock chains at night. No, no, no, no. I don't think so. I think that there are going to be a small number of people who want that and are going to do all the extra effort to make it happen. But they're going to be the exception. And the people who are doing that, they're going to are basically going to be part of what I'm calling sort of like the little tiny research and development cell. Whereas most people are going to take the training or the prescribed sort of program of like, this is what you need to do in order to work in the sock or work in information security. And that's what they're going to know. And it's going to become a very sort of constrained thing with different piece parts because it actually is a tremendous amount of work. Right. Right. And it's going to take a little bit of time to get it out to little, and this is going to sound more disparaging than I mean it, but little cogs that fit into a larger machine makes it so that we can distribute the work across a lot more people. I just don't think that there's a lot of incentive for organizations to fuel somebody who's a broad generalist. And the only reason why I had to do that, and some of the folks who were doing this starting 20 years ago, it was like, I was effective at whatever thing I did. So they just started throwing more stuff at me. They're like, we've got other swing tools Man up, hold on. That sounds so familiar, Chris. And I think you're right, actually. I think it's a healthy sign, though, because I think it is showing that the industry is maturing appropriately to have some division of labor and to have… Instead of one stop shop of smart guy, throw this over the fence. He knows that security stuff. Yeah. Yeah. need to grow out of that anyway so i think that's the right the right way to go yeah well i don't think we need to grow out of that i'm sorry john what are we going to say i was going to say dude
--- Page 22 ---
for what you've been working on with like the sock stuff kind of that theme that i see in everything you're talking about is the sock area is where it's kind of our farm for the next generation security badasses and we really i see so many socks out there we were just dealing with a vendor a couple of months ago and i asked a question for your sock analyst what do you do to train them and they're like um we have internal vendor training on our products and that was of how they took care of it and sometimes i don't feel like we're taking care of this crop of the next generation of security professionals we're basically just throwing them in and not giving them any training whatsoever we're treating them up like mushrooms throw them in the closet make it dark throw a bunch of shit on them and hopefully all right yeah i actually if you if you know again sorry i've referenced all these other topics that i've done but the educause one that i released that i actually have my advocated program for training security operations analysts and it's and it's not like here go take you know sans training or go take wild west you know go take this it's it's more about like what does training actually supposed to do for people and how are you going to actually approach that and what are you building as a program inside of your environment and everything that i'm advocating in security operations is this notion of like there's a lot of it that really needs to be engineered and repeatable and you know like cmmi three four kind of stuff but you cannot lose the capability to be adaptable because we are where this stuff comes to when things go wrong and if you do not give people the opportunity to practice dealing with novel circumstances they will suck when you throw them into a real one but you ask but you're also dealing with these these these these problems but we're going back to the sock everyone thinks that the sock is like a state machine right like you go in and you're making a widget like every day i check the alerts you go and you check the alerts you go and you check the alerts and even whenever you say whenever the shit hits the fan the sock analyst has got to be able to do that and you're right they stand up and they're like okay well sock analysts fix this they have no idea how to do any type of forensics any type of analysis whatsoever and they have to figure it out on their own but how much of that do you think is driven by people in the industry and companies in the industry that are constantly trying to sell like soar platforms and they're going to do full orchestration and automation of incident response and it's putting things inside
--- Page 23 ---
the heads of management but holy crap we can just completely automate and our way out of this when everybody that's on here pretty most pretty much knows that that's garbage you can't automate your way uh through a full security architecture at all i agree you know what i mean but that's why it's like you have to start with here's what i know how to do here are the things that are good candidates for automation and that's sort talk and that's what i'm going to do and that's what i'm i actually have this long spreadsheet and i talk about um candidates for automation in terms of your procedures and i actually provide guidance on how to prioritize various tasks for automation or orchestration and and basically and i'm paraphrasing it but the the summary is there's stuff if you have a low tolerance of errors and if you have something that you do a lot it's a good candidate for automation if there's a specific procedure that can be followed it's a good candidate for automation but i you know most people i ask them how do you make decisions about what you're going to orchestrate and automate they don't have a clue right and so that's a that's a fundamental problem of misunderstanding of what your sock actually is capable of doing All right, thank you so much for joining us. I look forward to hearing from you all. Thank you. Have a great day. Thank you so much, everybody. Thanks for listening to the Meeting and Calendar. I'm Alex Martino, and I hope that you have a great rest of the week.
Related Q&A
- Q: OverviewA: Presentation details for WWHF Hunting by Numbers