Methodologies for Analysis (with Christopher Crowley) - CISO Tradecraft

Document
Resource Thumbnail

This segment of "Methodologies for Analysis" with Christopher Crowley discusses the concept of methodology in cybersecurity, focusing on analysis of competing hypotheses (ACH) and decision theory. ACH involves brainstorming multiple hypotheses, gathering data to support or refute each, and scoring each hypothesis based on available data. Decision theory is also introduced as a methodology for identifying optimal decisions. The conversation also touches on the importance of having a disruptor or "Tenth Man" in a team, who can challenge the group's thinking and introduce new ideas.

Featured Clips

"The problem of misalignment of cybersecurity operations with an organization."
"It's probably why I'm as weird as I am today."
"Fix it well enough. And that's it. Yeah. And it's, it's good enough."

Full Transcript / Content

--- Page 1 --- Hello, and welcome to another episode of CISO Tradecraft, the podcast that provides you with the information, knowledge, and wisdom to be a more effective cybersecurity leader. My name is G. Mark Hardy. I'm your host for today, and we're going to be talking about methodologies for detecting breaches. And on our show today, I've got a good friend of mine, a man I deeply respect, Chris Crowley. Chris, welcome to the show. Thank you very much. Really appreciate you having me. Well, Chris, for folks who have not had a chance to run into you, tell us a little bit about your background. What are you doing now, and what did you do, and things like that? Yeah, so right now, I'm doing consulting. I do solo consulting gig through my company, Montance. I teach a class through that, and that class is on cybersecurity operations. So I think that's... For me, the... It's an expression of all of the effort that I've put in through my career to try to solve the problem of misalignment of cybersecurity operations with an organization. I teach that. I teach for SANS Institute. I do a lot of research. I tend to do talks at a lot of different cyber events. In the past, I've worked for government agencies, a number of Department of Defense in the U.S. I've worked in private organizations. I have... Basically across probably every market sector, I have worked in an organization either as an employee, a dedicated contractor, or a contractor as a consultant. So my background really started in computers when I was a kid. I grew up near Boston, Massachusetts, and my first job where it was like actually getting a paycheck, I mean, I did some stuff where I was doing mail merges for my mom's friend for marketing when I was... I was a kid on my Apple IIGS, but my first job, I actually started when I was 15 years old and I did Unix, specifically Ultrix systems administration and VMS systems administration primarily was there to run tape backup at the end of the day after the engineers had worked on stuff. So this was a bunch of geeks who had left BBN, Bolt, Baranak, and Newman. Maybe it's a company some people have heard of because they felt like it was getting too corporate in the 80s. Well, BBN, they were the folks who actually turned off the internet back in 1988 when the Morris Room came out. Because I remember they had the gigantic off switch that later governments that are totalitarian wanted one of those. Yeah, exactly. It's really interesting. Like BBN, I'm sorry, I don't want to talk about this too much, but some of the stuff that I talked to about, talked with those guys about is they would have stuff where they were doing network monitoring, and this is, again, in the 80s, this is exactly what we're talking about. They provided the internet to everybody in that area. --- Page 2 --- And so they would have, these were the systems administrators that I was talking to that were helped to start that. And so they were talking about stuff where it was like, yeah, when there was a network outage, they would figure out if it was a problem or not by basically calling the fax line that they knew was the fax line at that customer, and then verifying that the fax actually responded or not. If the fax didn't respond in a timely fashion, then they would. They would basically say, oh, there's probably a power outage in that location. So it's just like these very clever, intuitive things as a 15 year old kid dealing with a bunch of computer professionals at that time. So that's where I came from. Right. So. Yeah, pretty neat. So you got an early start and like a lot of us are probably in our teen years working on things far outside of the scope of what you'd expect. But back then it worked because there were, you know, nobody cared. Yeah, exactly. They didn't care whether you haven't been able to grow a mustache yet. It's just like. I mean, they were giving kids like me root on all the systems, which in retrospect they cared about, but they didn't really understand just how curious a 15 year old on a wide open network would be. So lots of NNTP and FTP time for me and lots of access to data. It's probably why I'm as weird as I am today. So that set the hook. And so let's fast forward a little bit and now we get to the point where we're in the corporate world. We're, we're supposed to hold down a job and look responsible and work a nine to five paycheck and commute and get a house with a fence and everything else like that. But it wasn't necessarily going to be your future. Was it? No. And I mean, I did that for a very long time and then ultimately decided that I wanted to be more self-directed. And the segue to that was really when I left New Orleans in 2005 because of Katrina, I had already been starting to move into that arena of SANS Institute and teaching for them based on my first experience with a 10 year old. Yeah. So I was attending a night session in New Orleans, probably in 2002 from a SANS event rolling through town. I was like, oh, this is really useful information. The first class that I took technically was Hal's Unix class where I basically had this, had this set of books and then I took intrusion detection because when I was working at Tulane University we were having problems. In the early 2000 era, that was when a lot of things started coming to the forefront of cybersecurity. So I established a anti-spam program. Because we needed that kind of thing where we're basically, we're able to stand up defenses --- Page 3 --- that didn't exist before. An interesting era of cybersecurity. Yeah. I remember doing stuff back in the day. I had a client, oh, what were they? We were using Novell NetWare. And so of course you come in that way, it was all dial up practically and they end up in the F drive. And I always told people the F drive and Novell NetWare. For those who weren't there was like that train station in the matrix where you just, you can't get out of it. You go around and you stay there. And at the time the question was, well, how do we make sure that somebody dialing into our system has updated antivirus? And it's like, well, okay, let's solve that problem because no one had solved it before. And I ended up writing a whole bunch of DOS batch files that would basically kick off and it would query it. And if the date would start that, then off we went and great. And now we come back and you look 15 years later and there's entire industries around being able to go ahead and do stuff like that. But like you, you just see a problem and you go, you fix it. And then in a lot of our cases, we just moved on to the next problem or hop on our bicycle and rode home. And then someone else. Or fix it well enough. Fix it well enough. And that's it. Yeah. And it's, it's good enough. I, one thing we were trading stories back and forth. So a different customer, they were doing CC mail. I don't know if you remember that back in the day. And what CC mail does is they're mail forwarding. So they call them routers. They weren't really routers. They were network sense, but I was working for a consulting firm and they did a firm fixed price bid for one of the statewide utilities in, in Maryland. So what happened was, is that we were going to run it on OS two, because that was going to be the really first multitasking operating system. And they said, Hey, we've got all this stuff that'll work there. And we figured, well, this is great. We can have seven of these things running and that will take place and we'll have all these different sessions. And then we'll turn out as vaporware is what we're supposed to run, but we're stuck on fixed price. Now I got to roll out 180 of these things for the price of eight or seven and we figured it out. And, but one of the problems was that you had to leave this thing setting out there at one of these stations all around there, these utility companies. And it's an unattended terminal with admin access. That doesn't sound like a recipe for long-term success. So I said, well, okay, well, what would I do if they had to solve this? Cause I got to, and we don't have a lot of money extra. --- Page 4 --- So I said, let's set up a little semaphore. So I had one machine that was running. Put it in their data center. And what would happen was you had a shared drive. And if you needed something done, like to do a file backup or a consolidation, which all of those required admin privilege, you just set a little flag in there, which is a text file with what I'm I'm station number 47 and I need this. And so once every minute, this thing would scan through, are there any files in that? Oh, there's one go do this, that the other thing, delete the file and off you go. And so what happened then is that the worst you could do. Yeah. You got a whole, one of these machines out there is you could trigger a privileged action that was supposed to happen anyway, like backup, but you couldn't have arbitrary command control. Okay. No big deal. Fast forward three years. I'm on my own. Now I get my contract with the same thing and I'm migrating their EDI system for windows into Linux. And so when I get into the data center, it's like, wait a minute. I remember I walked over and look at the screen. It was my program. It was still running three years later. It had not stopped because I found a little TSR for those who don't remember terminate and stay resident code that basically said, if you saw a C colon backslash greater than on the screen, it meant that somehow it popped out of the batch loop, reboot the machine. And that's what kept it back and going again. And it's just a little innovations like that, that you look at them from almost 20 years back and you go, how do we pull this stuff off? Because today let's face it. We become tool masters. Yeah. Yeah. If you're going to do something like that or detect a breach, you're not sitting there like it was in the early days, mano a mano, banging away on the keyboard, seeing who could get whomever. Well, it's an interesting construct. The other way to think about that is computer systems are specifically designed to be flexible information processing, and they're not designed to be dedicated task processing. So you build something with a little bit of resilience in it. Computer system. The information system is specifically dedicated. Yeah. designed to be able to absorb new and adaptive crops that you feed to it, which is exactly what you're talking about. I've got this general purpose thing, and if I make it just a little bit more flexible, then it likes that and it goes down that direction, which is actually one of the reasons why attackers are so difficult to remove from our systems, because everything in the system is designed to be flexible and adaptive. And if they're more flexible and they're more adaptive, then they will beat us on our own systems because we cannot keep up with --- Page 5 --- them. And that's a very good point because what happens also in our mindsets is that like any other trade, if you go back and you look at the traditional trade, they want to go ahead and go to trade school. I want to become a plumber, an electrician, a carpenter, something like that. I got to learn a basic set of skills. What's going to happen over time is I'm going to perfect those skills. I can get better and better at that. I can become ultimately a master in what I'm doing. And so there's a well-defined career path. But what we're happening is as we're watching this whole thing blossom and emerge and change around us, we don't know what's going to be there five or 10 years from now. If somebody told you two years ago that you're going to be doing work fighting a chat GPT automated attack tool, that's science fiction. And yet here we are playing around with this stuff. It's really interesting. I had a conversation decades ago with a microscopist because when I went to undergraduate, he was retired and he was working in a laboratory that I was basically a maintainer of, microscopy lab. And I remember having this conversation with him one time and I wanted to talk with him about some problems. And he said, well, what do you want to talk about? And I basically said, well, interpersonal stuff and politics. And he said, I'll come and talk to you, but we're not going to talk about politics and we're not going to talk about interpersonal relations. I'll teach you methodology because by teaching you methodology, that will actually serve you far greater than sort of this momentary concern that you have. So it's a, it's a lesson that I learned a long time ago. That was really nice in terms of what to focus on and the important aspects of things. Well, since you mentioned methodology and coincidentally, it doesn't happen to be the title of the episode, we might want to get to that after this long preamble. Hopefully everybody's still with us, but this is always fascinating. That's why I love having friends on the show because you'll learn about some amazing backgrounds and hopefully other people can learn about some of these things as well. And I think that's pretty cool. And I'd love to keep listening to this guy. So when we talk about methodology, let's kind of start well with the basics. Like what do we mean by methodology? Yeah. So methodology generally is, is an approach to problem solving. You, you want to fix something, you want to address an issue. It is what rigor will we apply to this so that we can be directed in our effort? Okay. And so what we have then is more of a strategy for problem solving as compared to well, I guess even randomly trying things, pushing buttons, like Homer Simpson, eeny, meeny, miny, moe, technically is going to be a methodology. It's not a recommended --- Page 6 --- I just push buttons until it's until it's actually working is a methodology. It's not our preferred methodology, but it is a methodology which some people can employ, or it's the only method that some people can employ as a way to say that. We like, we like other methods and it's something that I actually want to say, first of all, the reason why we would choose to have a methodology, which may be a foregone conclusion to a lot of people, but if we articulate it, it helps us to be able to decide what the right one is, is the reason why we have a methodology is to make sure that we haven't overlooked stuff and to make sure that a number of different people involved in a team can work together in an efficient way to come up with stable, reproducible, quality in the results that they're that they're tasked with in the face of novel circumstances, exactly what you're saying. We don't exactly know this. And so we're able to be adaptive and apply our methodology in circumstances that we maybe we hadn't thought about before, which is why we have the method to make sure that we address all of the concerns and have a stable point of reference for, for things in the future. We'll talk a little bit about some ideas and methodologies because of course, we say kind of the, the base case, which is, well, just push buttons till something works. And we say not preferred, but it does qualify as a methodology because you can define it and you could follow the process. And again, a methodology is not guaranteed to give you an answer that you like, right? You, you may come up with the saying that, Hey, my hypothesis was wrong and I, there's no way I can get there. So, well, so let's keep with this definition a little bit of that. What do we mean by a hypothesis? Okay. So, so hypothesis is a, is a, a structuring of a, of a statement around a circumstance. And usually the hypothesis is an expression of your thought of what might be a plausible explanation based on some, some data that you have. So let me, let me jump into a couple of things. So first of all, there are different techniques that we leverage in order to articulate hypotheses. So scientific method is one that every, everybody goes back to all the time. And a lot of times people talk about this in the context of cybersecurity. I don't like the discussion of scientific method in the context of cybersecurity, because in practice, we don't actually follow the scientific method, which would be an articulation of a hypothesis and then a construction of an experiment intended to disprove that hypothesis. And then if we disprove, that hypothesis, then we throw it away and we seek another hypothesis. And so it's this sort of iterative approach to a stable hypothesis, which eventually becomes a theorem because we haven't --- Page 7 --- been able to disprove it. And, and in practice, we don't actually work like that in cybersecurity because we don't have the time and the resources to be able to come up with these sort of more, more grand notions of, of theories. We usually have limited time. We have to come up with a best explanation of a theory. And that's, that's, that's the, that's the, that's the, that's the, that's the, of available data as opposed to a sort of more theoretical expansive expression of a a construct that allows us to work in that. That's not what we need in cyber. What we usually need in cyber is this is something that is intentional right now that is good enough for the purpose. And, and this is largely in the context of investigation. Now, if we're talking about detection, engineering, say then maybe, лов could do a different approach where it's more of an engineered approach and we might leverage some sort of a hypothesis construct to be able to say here's the thing that we're going to say we can do and now let's try to break it as much as possible until we can say look we've tried everything and we can't break it okay that's the one that's closer to scientific method than what we do in investigation and sort of more immediate responsive action so let me just continue to say one other thing I mentioned a methodology I don't like talking about scientific method I do like analysis of competing hypotheses from Richards J. Hauer in the Central Intelligence Agency book psychology of intelligence analysis he talks about this analysis of competing hypotheses and he kind of presents this methodology which I'm going to give you a very simplified picture of that he says you have some situation that you need to deal with you don't have full data you don't have clear proposition you need some revelation and you're parameters of accuracy, you need to come up with the best explanation. What do we do? We look at the data that we have, we brainstorm some potential explanations. We come up with an expansive list of potential explanations. Based on that, we go find the rest of the data that might be available to us. And then after we sort of reconcile the data that might be available to us, in cyber, we talk about this as visibility, right? We seek greater visibility. Then we have these hypotheses and we kind of consolidate and work on it. And then this is something that I've, almost never seen people in cybersecurity do. We have rigor in our assessment of the hypotheses in light of the data in a quantifiable way. And basically this is pretty simple. At its simplest, make a spreadsheet. Here are the columns of my hypotheses. Here's the data that I have. I'm gonna score it based on each data element for the specific hypothesis. And then I've articulated and in Howard's term, --- Page 8 --- externalized the construct of my investigation and my analysis, which allows me to do sensitivity assessment of what points of data do I really depend on? And it also allows me to compare between different analysts, what different people think. And I then have a way to express either my analysts agree or my analysts disagree. And if I have an individual assessments, that's an individual assessment that's performed between a number of different people. And I can do that comparison. If they're all sensitive to the same data elements and they all come up with the same conclusions based on the same data set and the same hypotheses, well, then I have very high confidence in the expression of that. Without having to go and spend a tremendous amount of time, I can do this relatively quickly. So it's a faster method which introduces its own challenges, but absolutely provides value in the sort of situations that we tend to encounter in cybersecurity. So when you talk about that and building that spreadsheet and saying, here's what my requirements are, here's the data, let's fit it. I think some folks might be thinking, okay, that sounds a little bit like the MITRE ATT&CK framework, right? Got tactics and techniques. And that might also be like a NIST 800-171 or an ISO 27,000. And I'm just going down the list, going down the list. That's not really what we're talking about. That's not what we're talking about. So you said two things. You said, here are my requirements and let's fit it. And it's actually far more open-ended than that. I will say that MITRE ATT&CK framework is also just a two-dimensional array, right? I mean, so it's like we use two-dimensional arrays a lot for our decision-making and it's a perfectly valid approach. It's a simple approach. We could add additional dimensions if we wanted to, but for the purpose of the sort of timeliness that we usually need, in an investigation and a response and a decision that we would make, two dimensions is gonna be good enough. But let me just sort of characterize this more intentionally for the terms that you use to make sure people understand it. Number one is we brainstorm hypotheses and the hypotheses would be columns in the spreadsheet. We identify available data and the data is the rows in the spreadsheet. And then the analyst's task is to assess based on some available quantity per row, a weighted ranked expression of which hypothesis is most supported based on that specific data element that's there. So this is the structure that he establishes. We're not fitting it to something that's an expectation. We're articulating our thinking on what we know based on the data that we have, and the hypotheses that we have agreed are within scope for this particular exercise of thought. And that's where it's gonna be the big difference between doing this compliance type of approach that I was talking about is that you have a set goal. --- Page 9 --- Nope, no goal. As compared to these types of methodologies where you say, I've got a situation, and I think this might be the answer to this or this, but you're gonna find out, and you're gonna be data-driven to determine what is really happening here. And flexible. And so the hypothesis, if you read Hauer's explanation of it, the hypothesis generation stage is intended to be incredibly flexible. And he talks about the things that analysts tend to do. As soon as they have a certain thought, that's the thought that they're stuck on, and they keep looking for data that matches the hypothesis that they've already arrived at as a foregone conclusion is the best explanation. And this is one of the reasons why he says you need to use this methodology, is if I say, okay, brainstorm all your explanations, and the analyst comes up with two, and it's A, and A with a slight variation, and I go back and look at the analyst's work, well, then I know that they've encountered this foregone conclusion problem. But if I go and look at the analyst's work, and they've come up with 20 different hypotheses, and then they work through the notion of consolidating this and finding other data elements based on the hypotheses that were added at the brainstorming phase, they've done a better job, and I have more confidence in the work that they did in terms of the analysis. So you wanna, you wanna get, you know, get into like a specific example? Because we're talking about this, and I think that this is a good one to help to illustrate it. Because I can talk about it theoretically, but we can do a very specific one. All right, so yeah, one last thought on that one. I was on a call last night, and someone was mentioning that he was working at a high-level organization in the government, and he said when they came up with a idea or whatever, and when everybody thought it was a great idea, it was almost universally wrong. It was a combination of groupthink, a lack of testing the hypothesis, a lack of rigor. And so he had learned as a senior executive is that when you bring something in and everybody says yes, you need a disruptor. You need to find some way to say, okay, push the reset button and then re-control all, delete your group, and said no. And so if we look at the history books, my understanding was that was what Bobby Kennedy's role was back in the Cuban Missile Crisis. There's a team role name for that, and I think it's the Tenth Man or something like that, where it's like there is a role that the only thing that you're allowed to do is disagree with everything. Right, and I mean, some people I know, then that's their just M.O. all day long. You know, we have other names for that, --- Page 10 --- but I can do that. Don't always invite them to the party, but yeah, exactly, yeah. From the notion of being stuck in the, oh, well, that must be it, because those smart people are addressing it, then yeah, having somebody that is the disruptor, is quite valuable. But just to that end as well, that disruptor would say, okay, no, that's not the hypothesis. No, that's not the hypothesis. We don't want that. We want, let's get all of the things, and the disruptor's role in the hypothesis brainstorming would be, here's another one. You haven't thought about aliens from Mars. You haven't thought about zombie apocalypse, or whatever it is, just these sort of like real outside expressions to help to break people out of the, the, the, the constraint of non-creative thinking. That's really important as a leader. I think I mentioned it one or two shows ago about I put an officer on my leadership team who always saw things differently than I did. And I told her, I said, you're gonna have the toughest job here because you're gonna be outvoted nine out of 10 times, but one out of 10 times, you're gonna save the show. You're gonna come up with something that everybody else did not think of. And because we have a respectful exchange of ideas, and you bring this thing out, and everybody's gonna go, whoa, wow. But as they say, when you're nine out of 10 times, you're getting voted off the island, it's tough emotionally. And so for those disruptors or whatever title we give them, you have to also make sure they've got the resilience to be able to interact in a way where you know that your one loss record is always gonna be very terrible, but when you win, it's a decisive win. I mean, you strike out nine out of 10 times, but then you hit a grand slam. You strike out nine more times, hit a grand slam. It's a grand slam. It's a grand slam. And the expression of analysis, though, we don't, so to me, in my thinking, I don't wanna talk about win or lose because in these sorts of questions, we will never know. We will never know right or wrong. It's not a, did this happen or did that happen? It's a, there's something that happened, and we're trying to explain it, and we will never know authoritatively if we are right or wrong. That's just the circumstance that we find ourselves in, and that's why Howard says we need to have this structure. Okay, and so again, this is one. There, let's, one other thing I do wanna mention, it's not one that I use a lot, but it's one that I've been looking at more is decision theory. And so it is a methodology of being able to actually identify optimal decision. And I still don't think that that's right for cybersecurity operations, but it's just one other one. --- Page 11 --- If someone's hearing this, and they're like, oh, okay, well, let me go get into these different methodologies, comparing and contrasting decision theory from analysis, beating hypotheses from scientific method is absolutely a valuable thing to do. So you can pick the right aspect of what you want your staff to do. And you mentioned MITRE ATT&CK framework, also really good for things, investigation, response, detection engineering, defensive gap analysis, all sorts of things you can do with that too. But I feel like we're more in the kind of like the hunting investigation kind of mindset when we use ACH. So let's take a threat hunting type of an example. Let's say there's been a potential for a breach and you're trying to figure out, we're concerned that maybe somebody has exfilled data from our organization. Maybe we use that as our null hypothesis. And so from that, we'll say, okay, one of them could be they're using DNS. And so we'll use our favorite port 53. So let's start that. For example, let's say that was on the table and you say, well, we're, they're getting stuff out, but we're not sure how they're getting it out, but we're pretty sure that there's something going on. So what would be a methodology approach that we could use being a little bit more specific, looking at DNS so that we can talk about it from a threat hunting perspective? Sure, so part of, we're sort of combining the structures of threat hunting as an act, and then the explanation of, hey, there's a problem, right? We have some clue that an attacker has taken things out. We need to investigate that. And one of the potential explanations is that they're using a DNS command and control to be able to do that. And so now we're looking for data that would assist us in investigating that. So now we need visibility aspects of where does that data live? And immediately you mentioned UDP 53. So we could look at, you know, we could look at, you know, we could look at network traffic on UDP 53 and the TCP 53, because zone transfers, and it doesn't have to be zone transfer. It can still run over TCP 53. We would look for- I think it was over a thousand bytes or something like that, it goes- Yeah, exactly. We would look for data, we would look for data on the DNS servers. We should also look for data on the hosts themselves. There might be syslogging, I'm sorry, Windows event logging, or all of these different aspects. We could look at cache data on the host. So now we have all these different potential data sources. By the way, there are two other, there are multiple other DNS protocols, but there are two other ones that we should be concerned about with respect to exfiltration of data. That's DNS over HTTPS and DNS over TLS. So DNS over TLS is going to be port 853 --- Page 12 --- that we need to concern ourself with. DNS over HTTPS is actually over port 443 or any port that you want it to be. And it looks like a fully normal HTTPS. So it's going to look like a fully normal HTTPS connection. So all of these are places that now we need to consider if there's data moving through them. Then as we go and gain visibility into the data that's available to us, then we would look for attributes within that data for elements that help us to make a decision. This is benign and I can remove it from my view and investigation. This is suspicious and I should continue to characterize it, or this is malicious, and now I have a problem. And so the expression there in our threat hunting is interesting for most analysts because, again, they're tuned to, we are, not just analysts, but human beings are tuned to, soon as you see something that kind of matches a pattern, you're going to click into that pattern and make sure that you articulate very quickly, I'm right. So let's talk about- And bias, I think, is a psychological term for it. Let's talk about some of the elements that we typically use in our DNS investigation in order to kind of assess if this is malicious or suspicious. So one thing that we would do with this is say, okay, let's look at history and look at all the things that we've looked at previously and say that that's automatically benign. Is that 100% accurate? No. Is it useful? Absolutely. So I could just basically say the domains that I have seen in my environment, 14 days in previous, 28 days in previous, 180 days in previous, let me just exclude those. Now, you would need to have either a table that lets you do that quickly, or you would need to build that table for exclusion. Actually, quick aside, I wrote a script that basically did this a decade ago. I wrote a script that basically did this a decade ago. To deal with DNS as a basically a vector of indication that there's a problem. And one thing that I did was I just said, let me record everything every single day in terms of what we're seeing. And then on any day where I have something new, a domain which I have never talked to before, let me look at it. And in a really big network that gets noisy, but there's still value to it because you could do that plus additional characteristics. So, what would those additional characteristics be? Very high volume of communication, long host names, weird response records, lots of invalid for a period of time followed by lots of valid for a period of time. And maybe even sort of a switching back and forth in terms of responses in DNS. We would call it an NX domain, basically failed to resolve. And then all of a sudden the response is --- Page 13 --- successfully resolved and then maybe flapping back and forth. back and forth between them is a weird behavior, right? Re-registration of domains to new authoritative name servers happens, but generally doesn't happen a lot in terms of domains that are in place, but this might be an indication of some sort of a domain takeover, maybe even for a short period of time. So just some thoughts about what an analyst would then go use as that preliminary differentiator. Now, in the methodology, in the methodical approach that Howard describes, all of that would be included, all of those data elements, anything that I'm thinking about, let's pull all of those into my data attributes. And still, hypotheses thus far were kind of DNS, DNS command and control, DNS for exfil. And we could actually, if we wanted to stay within the DNS range, we could say DNS over HTTPS, DNS over TLS, DNS command and control using a number of different tools. And we could actually just have a very detailed and granular set of hypotheses within the specific hypothesis of DNS in order to help to weight what we're doing. So part of this investigation is let's go find more data. Yeah. Right. And so what we have then is all these different hypotheses would be, if you will, our columns there in the spreadsheet where we're saying, okay, this has got a lot of weird stuff. This is a very long query. This has not an alpha numeric characters in the DNS query. When you're constrained to that, a lot of fails, and then it starts to succeed, and on and on and on. And then we just fit the data to the hypothesis rather than fit the hypothesis to the data, or which is, what are we doing? Well, let me say, instead of fitting- As soon as I said that word, I said, okay, this is gonna- Yeah, yeah. So let me say, okay, let's presume, we did our brainstorming, we've got a whole bunch of hypotheses. We've worked through this data collection for a while. Now, the data elements, all sorts of them, all sorts of data elements are in rows. Now what I want an analyst to do is to consider each row, each data element, in turn, as independently as that human's brain can isolate that data element, I want them to take a number of points. And let's say that we have five hypotheses. So I'm gonna say, per row, each analyst is allowed to assign a maximum of 20 points per row. And I want the analyst to grade the hypothesis on a weighted basis for each data row. And then each analyst should take each data row in turn and say, does this hypothesis, explain this at the absolute best to the exclusion of all the other hypotheses? Okay, that hypothesis gets 20 points in that row. And the other hypotheses get zero points. Now, if hypothesis A is not feasible --- Page 14 --- based on that data element, but hypothesis B is kind of feasible, and it's about as equivalent as hypothesis C, but hypothesis D is really good, and hypothesis F is really not feasible, then my point is, hypothesis D is really good, and hypothesis F is really not feasible, and hypothesis F is really not feasible, then my point is, if I had 20 points in that row would be, B would get five, C would get five, because I think D is really the best explanation, it would get 10. So now it's zero, five, five, 10. And if I do this in turn, and I'm basically just like ranking it based on my thought of that data element explaining that particular hypothesis, as I do that in turn, at the bottom, my summation function is going to say, Chris thinks, B is the, is the most weighted explanation, but he also thinks that C and D are probable explanations and A and F, not really. Now, when you mentioned that you put a zero in there, cause you say it can't happen to me, that just adds up, but it might get overweight by other things that could happen. So in doing that approach, would I ever get something where I would put a big, like a negative infinity in there and just simply say, you know what, that disqualifies us because it's utterly impossible for that hypothesis to have happened with this fact set. That actually, according to Hauer's method, and I didn't go into all the details, you would have excluded that hypothesis at the consolidation phase before you were even doing this grading. But sure, maybe, maybe at some point you're like, no, actually this irrevocably refutes this, but that's not the, that's not the expression. Typically it's non-negative numbers, right? It's like the constraint in the way that I'm describing it. But, but certainly there are exclusion that happened in our, in our generation of, of our hypotheses to where we, he calls it refining the matrix. You remove stuff, you knock it out, and maybe you have to go back and adjust the matrix after you're doing the analysis and you, and you knock it out later. Yep. Got it. So essentially what we're talking about then is having a methodology. We have some competing hypotheses. We take and collect the data. We then use the data and we score, if you will, that data. And then we take and collect the data and we score, if you will, that data. Hypotheses upon, does this hypothesis account for this? Does it account for it extraordinarily well? Does it could happen or no, that's not. And, and although I mentioned the zero or negative infinity, zero may be more powerful because you might say, no, this hypothesis or this data doesn't account for this hypothesis, but it doesn't necessarily totally disqualify it. Yes. Yes, exactly. Exactly. And then, and then think about reporting this. Think about reporting --- Page 15 --- this where you can actually now articulate your confidence in a specific explanation and also express your confidence in other explanations, which you have considered. And, and the reporting becomes a lot more robust in, in, in my opinion. And, and I think for people who would be hearing that, you would say, we considered this, this, and this, and this, and our analysts assess their confidence. And you could also, if you wanted to get fancy, with the math, you could also show the distribution from a specific explanation between multiple different analysts who have assessed the same hypotheses in light of the same data. And we actually, you remember this advertisement, four out of five dentists agree, which is exactly what you were talking about earlier. Nine out of 10 of my, my officers agree. But think about that. If rather than it's okay, well, you got bounced out because you, you didn't. Now, if everybody had, you know, you're not going to be able to, you know, you're not going to be able to, you're not going to be able to be able to articulate their work in expressing the, the why, the why, the why I think this, and this is exactly what Hauer says we need this for, is it because it externalizes that? Because some analysts are just going to yell louder than others. And some analysts are going to keep talking until nobody else is willing to argue against that analyst because they just don't take a breath. And, and you've got to also think of the group dynamics, because if you've got somebody who's new, and they come up and said, Hey, I've got an idea. And one of your old Kerbergen said, well, that's the dumbest thing I ever heard of you, moron, or whatever, it's going to shrink down. But it's like, he's never going to speak up again. It crushes people's drive to, to express, especially people who either don't have confidence or just want to be analytical and they don't want to fight the verbal argument. This actually helps to equalize that. And the other thing that occurs to me is we've talked this through, and we're pretty close to the end of the show here. And it feels like we just go on forever, but we could go on forever. In fact, we're going to bring it back on the show. There's no, there's no question about it. Let's do a strategic view in the future. I do to this because we have multiple analysts and we're looking at the same data and we're scoring different hypotheses. But over time, what I would do as a SOC leader or as a CISO is I was going to say, you know what? This person's got a better track record than that person than this one. It doesn't necessarily mean I'm going to use it for hiring or firing decisions, but I might put a little bit more weight on the person who gets it right more often. --- Page 16 --- You could do you could do individual weighting. One other thing that you haven't mentioned, but I absolutely think that you would do in a SOC is you would actually automate and orchestrate a number of these tasks already, because if we keep having to go get that data, if we keep building hypotheses through brainstorming, actually we can we can expedite a lot of this stuff, which is the context that I came up with use of analysis of competing hypotheses originally was actually. In an orchestration and automation context in order to help us to move faster and eliminate certain things that are repeated, but still maintain good analysis, still maintain the human being. And I can actually express where I want the human being in the process of doing the ACH and where I want the system automating the task. It almost sounds like we could then go ahead. And if we're looking at competing SOAR tools that we could apply our methodology to say, hmm, which one of us gives us the. Injects when we want the injects is compared to, it just automatically goes past that. I had that conversation yesterday with a MSSP that they had an automate solution that they had. And I said, I don't get a chance to trigger my automated playbook until after your human analyst has made a decision to say, yeah, this looks genuine. I said, if I'm in a situation where I think I'm under attack, I'll take some false positives and I'll put them on my table. So I'll look at them at the same time you look at them at your table because you might have five other clients who are lighting up a switchboard. And so. I need a bypass up and over that last step. And they said, I've been asking for that for over a year. And I said, yeah, we're built. We just built that. So sometimes if you're a squeaky wheel, you can get vendors to, to adjust to what you're doing. Any, any last thoughts before we wrap up on this? Just really appreciate the folks considering what I've said and thinking about how to fold that into their environment. I think it's an important aspect of people doing good cyber work. Well, that's awesome. So Chris Crowley, thank you very much for being part of the show. We'll put links to how people to find you on Montance. M O N T A N C O. E dot com into our show notes, as well as some links to some of these methodologies. For those of us who are just listening to us on the podcast, Hey, tune into YouTube. We're starting doing that at the beginning of this year, and we're going to make other short videos available soon. So subscribe over there as well, so that we can go ahead and help people get the word out. But this is CISO Tradecraft and until next time, stay safe out there.