2026 SOC Survey Jupyterlab and DEIDENT Response Release (Video)

--- Page 1 --- All right. I'm going to start up in a minute. It's about that time. Thank you for coming. My name is Chris Crowley. This is a talk on a paper that I've written. I've been writing this paper for about 10 years. This is security operations and sort of the resources around that. I've been writing this survey through SANS Institute. And the intention of it originally was to help people to understand what they should be doing by comparing to their peers of what their peers are doing. A lot of people want to understand what organizations do for their security operations, but they have a tough time asking the right questions, figuring out where people are at. There's sort of this informal sharing that occurs, but this was something that I started writing in 2017, asking hundreds of questions. And it has been something that a lot of people have come back to me and said, hey, I took this, took it to my management, explained what we saw that other people were doing, and it was quite useful for us. So I continued doing that. This year, there were over 400 people, 54 different locations that were basically in place for the respondents that they were actually physically located in. There were three different parts to it. So there was a cyber leader, so leadership part, asking a small number of specific questions. There was a shorter part for people to do the survey, and then a longer part of the survey that was extended for this. I actually use a couple different things to do the analysis of the responses that come back. Some of it is a spreadsheet that SANS generates out of the Qualtrics questioning. And then there's also a JupyterLab notebook that I use in order to do analysis. And I'm going to share a bunch of the details on that JupyterLab notebook. notebook tonight after I go through some of the slides. And then I also tonight, I'll probably actually get to it tomorrow, but I'll release that Jupiter Lab and the de-identified data, which is something that I've done since 2020. The point of sharing that information is to enable people to answer their own questions. Starting in 2018, I had somebody message me and say, hey, look, I work for an enormous financial services company. I don't care what people in the education sector do. The only thing I care about is what other big banks are doing. He's like, can you help me to understand that so I have details so that I can actually do a comparative analysis between me and my organization and what other organizations that are like me are like? And he told me some details and I was like, well, no, I actually can't. I can't do that. I can't do that right now, but let me see what I can do to help with that. And so I started pestering SANS, like, hey, I need to release all this data. And they're like, oh, no, you don't. --- Page 2 --- And so eventually I won. And in 2020, when I did that, because actually SANS didn't run the survey in 2020. I ran it independently. I just did it on my own. I only got 150 people in 2020 or something like that. But still, it was pretty good turnout for me just doing it without any sponsorship. So I just went ahead and did that. And I took the data and released the de-identified data. So I've been doing that since 2020. And what I've tried to do over the years is keep the questions exactly the same so that I can do comparative analysis. And this is something that I'll actually get to in the JupyterLab notebook, where I actually did a year-over-year comparison on a specific question. And you can see some pretty lines in there and stuff. This is actually really hard to do. You know, it's like if you're doing any sort of data analysis within your organization, imagine taking the data from different years and then trying to stitch together that stuff and say, which of these are actually the same? Because people keep changing stuff. And then being able to do that correlation is actually a challenge. So I really have insisted on sort of freezing the questions for a while. Still hasn't actually worked exactly, but it's done pretty good. And on the other hand Iいつ where you have a bunch of a thousand different questions and of those questions over these years, really only about 20 of them are exactly the same. So, another thing that I have done multiple times is I take this sort of like cohort approach. One is, and this is one that I did, where it's like, there are a bunch of people that said, you know, what, we don't even bother trying to decide what goes into the sem, we just stuff it all in the sem. We just don't do any pre-filtering decision or whatever. We jam all the data in the sim, and then we just use it for stuff. And my question was like, okay, this is a pretty big choice in terms of organizationally with your SOC, what you're going to do. You don't decide specifically, you just jam it all in there. Does that make people better? Do they have more capabilities? Do they have greater technology satisfaction? So I took this cohort type approach, and the code that I have for this cohort approach, you could basically take that on any sort of dichotomy within the data and then repeat that and say, okay, given this A, B comparison, can I then cascade out and see what things, you know, the people who say that they hate the sim, what does that look like for them on the other things? So this, again, is code that I will share eventually. You know, within the next couple of days, I'll finalize it. I also started doing this from a comparative analysis to be able to do the different feature analysis on the cohorts. --- Page 3 --- So I have a bunch of different things that I worked through to try to find what actually has statistical significance for these different cohort components. So that was an interesting thing, and I'll talk about that a little bit more. Okay. I also, you know, have seen this year after year in terms of the capabilities of what people say that they can do. So the capabilities within the SOC, there's some sort of alerting monitoring. There's some sort of security administration, architecture for IT, and incident response. So this capabilities IR is incident response. The people that answer it, they tend to be in cybersecurity, banking and finance, technology, and government. This is sort of the standard competition. This is the standard composition for SANS students, and it's the standard composition for the people responding to the survey. Why? Because these are the people who are spending the money to try to protect their information assets. Like, this is the core composition. There's not a lot of, like, manufacturing represented currently. Why? Because manufacturing is just kind of like, we don't really, you know. I mean, there is manufacturing, you know, sector people responding to the survey. But it's not. It's not as urgent of a problem for them in terms of cybersecurity. Of course, it's largely North America. And most of the people responding to the core survey were analysts, SOC analysts. The other part of this is that in addition to that core survey, there was also an executive-focused survey that lifted, like, five questions out of the main survey. And then we did kind of a comparison of, like, what are the executives saying and what are the analysts saying? Which surfaced some pretty interesting things. As far as the executive perspective, visibility seems to be the problem. There's a concept that the management thinks that they're doing a great job, like, being concerned about the hiring needs of the SOC. But the SOC doesn't agree with that. Like, there's a perspective that management doesn't really care. They don't really care about the things that the SOC needs in terms of staffing and staff roles that are there. Cyber threat intelligence is used consistently, but it's really not something that's used to help to inform the budgeting. And, you know, the biggest thing that continues to be a problem is the concept of human capital. There simply aren't enough skilled, trained, effective people to help to address cybersecurity. And, you know, you hear this. I hear this expressed in a lot of different ways. People say we just spend all of our time keeping the lights on. You know, this concept of, like, we just make the systems run, but we don't have a lot of time for analysis. We don't have a lot of time for improving the security posture of the organization. And so these are some, you know, sort of things in there. You can read the report itself. I'm just trying to hit a couple of salient points before I get into some of the code. --- Page 4 --- And stuff that I was working on for this AIML. The question that we asked about it is, is it operationally integrated into the way that you're running cybersecurity operations? And what what most people said was actually, no, we're using them. We're using the tools. But what's happening is we actually are not integrating that into our workflow. I think that this is going to change. Over the next couple of years. But most of the people that I work with from a consulting perspective, they haven't figured out how to do it yet. It's all still kind of ad hoc individual. I like to use chat GPT for X, and that's what individuals are doing. They're not actually saying this is how it runs within the environment in a in a dedicated way. Another thing that is always interesting to me is what makes people stay. I mean, because cybersecurity, if you are not aware of this, it's kind of a burnout field. It's pretty easy to just be like, all right, finished here. The other part of that is not only is it a burnout field, but there's actually almost always another company hiring that you can just flip over to. And, you know, there are different statistics in terms of retention, but especially the like tier one kind of role. Where it's just you have to make decisions really fast and you usually don't have an abundance of knowledge and experience. After a year in an organization where people are frustrated by their management, they're just gone and somebody else is going to hire them for five or ten thousand dollars more. And they're out. They're out the door. This is this is a common problem. But in terms of what makes people stay and this I've tracked this year over year. Meaningful work was the top one. And this meaningful work, career progression, money and training are always at the very top. And over the last couple of years, money has actually dropped down into positions like three and four. Now, it may just be the people who are answering this survey who are already within their mid career journey sort of thing. But they're like, I'm not going to change for a small amount of money because it's more important to me to care about what I'm doing. And why I'm doing stuff. Now, this is from people who are working in a sock saying that this is what I care about. So if you're trying to retain people, then you should actually make sure that you're you're providing them with something that's that's, you know, reasonable for them to care about. One to three and three to five years in terms of the in terms of the retention. This is a thing that is actually there. We consistently have seen. Three to 10 FTEs as the size of the sock. And it's interesting. I've in previous years, I spent a lot more time digging into this adjusted for organizational size. --- Page 5 --- Sometimes it gets bigger. Most of the time, even at the at the bigger size, it's three to 10. And what I think happens with this is that the organization as the organization gets bigger and more mature in their security operations, they outsource more and more stuff. They just shed that and they have these internal FTEs and people aren't counting that the SIM is managed by the third party, that the tier one triage review is managed by the third party, that the tooling is managed by the third party, that we also have a managed detection and response capability. And they're not counting that into that. But then you have these FTEs internally that one person's managing the MDR relationship. One person's managing the SIM relationship to the vendor who's helping with that. So I think that you have this sort of number of. Three to 10, you know, constantly reoccurring. And that's why. I asked a little bit about how do we hire people and what do we care about and which skills are difficult to hire for. And it's not always about being able to run the technology. In terms of finding people that are good matches for security operations and knowledge management. And organizational awareness and business acumen are things that are at the top of the list of non-technical things that are there. From a technical skill, the idea of doing threat analysis and data analysis, very interesting. This is something that if you can actually assess what the threat is, you can help the organization to deploy resources to counter that threat. And they're not wasting time and money and resources on things that. Aren't viable threats to the organization. So this is what people were asking for in terms of in terms of recognition. I talked about that a little bit. This year, like previous years, I've graded the technology satisfaction on an A through F scale. This year is, you know, a GPA scale. Nothing reached 3.0 or above. So the the highest satisfaction. Satisfaction was for the endpoint or extended detection and response type tools, the two point eight nine that's been the same for the last three years. That's been the top tool most people like their EDRs. An interesting thing about that. Is that from a hiring perspective, when asking about the technology for hiring, the sim is the more important technology for hiring, as reported by people in the survey, the EDR is lower. And my. Interpretation of that is like people feel like the EDR is taking care of itself, you know, like the EDR is doing its thing. We don't need as much sort of like technical acumen for the EDR. The sim requires human oversight, intervention, tuning, adjustment. And that's why I think that people are people are actually hiring for sim technical capability. Metrics are pretty important to me, man. I forgot. I can't believe I forgot this. I brought books to hand out because I just wrote a book on metrics and I have them upstairs. --- Page 6 --- So at the end of this, if you all stick around, I'll run upstairs and grab some books and hand them out. But but basically the the idea here is that. I wanted to understand what people are doing from their metrics program, because the metrics are typically what we use in order to say, am I doing a good job? The number one. Metric reported that people use number of incident handled. I. Hate this metric because it doesn't indicate whether you're doing a good job or not. How many incidents did you handle last month? Thirty eight. How many incidents did you handle the month before that? Twenty four. I don't know. Are you getting better or are you getting worse? So here's the thing. How many incidents should you handle? All of the ones that occurred in your environment. How do you know how many occurred? There's no basis. There's no way to compare that to what's reasonable. You don't have confidence. More incidents? Does your management want you to have more security incidents? No, they don't. But if you have more incidents, to me, what that means is you're doing a better job finding stuff. That's what that means to me, but your management doesn't want to hear that. Okay. So with this, I've been doing this for 10 years, and this is actually going to be the last year that I do this survey in this form. Next year, we're not doing it like this again, and it's going to stop from where it is, which is fine. One thing that I've noticed, is there's a bunch of stuff that just hasn't changed very much. Same problems, persistent, same issues, same staffing challenges, same metrics challenges, same technology challenges. I don't think it's getting better next year. I don't think it's getting better the year after that. I think that this is kind of the thing that we have, is we have this dynamic tension of problems, and we have what we have at our disposal. I don't think AI is fixing it. Anytime soon. What I think AI is doing, I'm going to take a brief digression into that. What I think AI is doing, especially over the next couple of years, is crushing your vulnerability management program. I think it is going to absolutely increase the demand for high volume, rapid deployment of critical patching. So if you're not ready for that right now, you need to change. That's hugely important over the next 12 to 24 months. And if you can absorb that, then probably your posture will get better. I think that the issues here that we don't have technology that solves all of our problems. We don't have the people to use the technology that we do have effectively. And we don't have the organizational support to be able to actually change the posture of the organization. So we're going to be in this constant fire drill. And I don't think that that's changing --- Page 7 --- anytime soon. So that's kind of the summary of the results. I do want to talk a little bit about the notebook stuff. So I'm going to show you some things inside of here. And then I'm actually going to switch to the notebook itself. And to the best of my ability, I'll answer questions on the fly while I'm doing this stuff. But I want to start with a little bit about how you would set this up if you want to use the notebook. Okay. Now, in order to do that, the way that I typically do it is set up an anaconda environment, which is a way to have a container of installation within a specific Python environment. I use the 3.12 Python for this. I create an environment called survey and then activate the survey. And the packages that I use inside of this are Seaborn, Matplotlib, Pandas, OpenPy, Excel. So this is for Excel spreadsheets, JupyterLab to provide the interactive environment, and SciPy for some of the statistical analysis stuff that I do. After you install this in the environment, you can run JupyterLab in the folder where the stuff is. And then you get this JupyterLab environment that you can work in. I actually check the data that gets released because this gets released publicly, open source, you know, Creative Commons, share alike, whatever. I would hate to take somebody's private information that they answered in the survey and put that out on the internet. I don't want people's names or email addresses and other stuff. So there are a couple of layers of validation and I actually have a notebook. That I do the de-identification validation to make sure that there's not stuff in the text fields that's like, you know, this is what my company does and this is the internal IP address and these are the ways that we're doing things. I don't want any of that stuff inside of that. So I review that. The next thing that I do when I start this off, and I've been doing this for years, I actually take, I have code to do this to help me. I take every single question and I relabel it in a dictionary. To be, to have a meaningful column name. And I do this as sort of like a quality validation thing. Part of it is to end up with a, you know, a dictionary that makes sense and I can use these shortened, you know, sort of categorized names. But the other part of it is that I make sure I understand everything that's in there. And for me to be effective at doing the data analysis, understanding what all the little piece parts are is useful. Good. I then, did a thing where I was looking at how long it took for people to, to finish it. And this is just a little snip from some stuff that's there. So in terms of, I split it into a histogram and put it --- Page 8 --- in 10 bins in terms of the time to complete. And I had one outlier, this one outlier. And what I think happened is the person started taking the survey and it was like, I can't remember what the exact number was, but it was like 27 hours or something like that. I was like, that seems weird. And what I think happened is they started the survey. They just left the, they just left the tab open and they finished it the next day. And so Qualtrics basically said 27 hours to complete. So I just got rid of that one. Like that doesn't count. But after I trimmed that one off and I did a, you know, trim the top and trim the bottom one, it was still an average time to complete. Of 47 minutes. People took on average, 47 minutes to answer all the questions. When was the last time you spent an hour answering a survey? Nobody does that. Nobody does that. So the other thing that I'm pointing out with this is that it's like, it's actually pretty interesting, the amount of time that people invest into this particular survey. And this has been consistent for years in terms of what, what people do. And I think that's what people do. So there is a lot of, there is a lot of value and thought that, that goes into this. Okay. So I looked a little bit about some of the, some of the things for this. I did an AB test and I was looking at different things that are out there, the threat analysis and the, the business acumen as things that, that are hired for. I took a look at some metrics, the way that we do metrics, we have metrics reported. And we asked the question, do you use this? If you use it, do you also enforce the metrics? So it becomes like a service level objective. And if you do have it as a service level objective, do you meet the objective consistently? So these are the different color coding things that are there, but then I've sorted this based on, you know, the, the top most popular ones. We saw that a little bit. I have the different color coding things that are there, but then I've sorted this based on, you know, the technology stuff. And this to me, it's a very, you can't read it, but it's a very complicated graph. I'll show you this line because this is the, this is a much easier thing to see. And I'll explain what this looks like. I have two different Y axes. One is the GPA of the technology. I just removed the names of the technology because it doesn't really matter. On the left-hand side, it starts at the, you know, the, the technology rating and goes from, you know, the technology, basically list of things. And you can see the peaks of the GPA. And then the blue line --- Page 9 --- represents the production deployment. Did you get to full production deployment? Okay. And you can see for the things that got to full production deployment, the GPA is generally higher. And this to me is, is an interesting thing that I see. And I've seen it year after year after year. And I don't think that it's, it's causal. I don't think that like getting to production deployment makes you like the technology better, but I do see that there's a correlation here in terms of like the GPA tends to trail off as you get to you know, basically a lesser and lesser production deployment. And my guidance from this around people and what they're doing with technology, what I've seen is that the GPA tends to trail off as you get to, you know, a lesser and lesser production deployment. And so what I tell people is you really need to have a clear plan. If you're going to acquire new technology of how you're going to get it completely implemented. This to me is like a catastrophic failure that I watch over and over and over again in security operations. You have a team, they decide they need a technology for any number of reasons. They decide they need a technology. They spend all this time and effort, going and getting budget for that technology. They spend the money and then they get halfway through the deployment. And it's either not what they wanted, not what they thought it was going to be, or they run out of energy or steam or resources because the person that was supposed to do it, or the person that knew it best is gone, or because their management changed direction on them, or because, you know, there's this long list of reasons why you don't get to the full production deployment. And so I think that's, I think that's, I think that's, I think that's, but it's, it's crazy to me that people plan not to actually finish the thing completely, or they fail to plan how they're going to finish the thing completely. And to me, this results in dissatisfaction with the technology. Like continuous improvement too, right? So if you deploy something, you get it working too, you continue to get it to keep it in operation. Yeah. So the, the continuous deployment, continuous improvement, that sort of like CICD approach, I think that that would be phenomenal for a security operations center to embrace the CICD approach to doing things. In my experience, security operations is far less mature at that sort of like approach to deployments than our IT teams are. And, and here's why I think that is. I, I think cybersecurity operations is the entity that catches all of the unexpected things. IT, well, they have all sorts of challenges and stuff. IT is you have a thing, you have a project, you get it deployed, and then you operate it once it's deployed. Cyber is you have a thing, --- Page 10 --- you set it up, stuff changes, new challenges emerge, people have emergencies that you need to address, and you can't do anything about it. And so I think that's, I think that's, I think that's, you have to catch and adjust to all of these different things that are dynamic. And actually, I think that that's the right role for us. I think that it's the right role to actually be cybersecurity operations of embracing uncertainty and being the team that handles it. But because of that paradigm of addressing uncertainty all the time, we don't think in terms of stability and continuous improvement. That's not our operating paradigm. Our operating paradigm tends to be just in time. And so I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, I think that's, deployment from an emergency incident handling responsive approach. It's not a, I've got this stable base, and then I can adjust as necessary. It's, these are the things that are there, don't touch those, like don't change that stuff. And we're going to have all this other stuff over here that we're just like, you know, working on. Cybersecurity, completely caught off guard as a generalization. With AI technology. Completely caught off guard. You know why? Because as an operating paradigm, we don't do research and development on emerging technology. What SOC actually has a responsible person or team, or even, you know, a time slice of the people that are there to look at emerging technology and approach it? No SOC does that. Why not? Because they're too busy. And I'm putting that in air quotes for emphasis, because they're not too busy. They're just misaligned with what the reality of the environment that they're operating in is, right? And that's just the way that it works all the time. We're, we're behind on all these other things. And so we don't think about how do I start to get in a position to get ahead? And the CICD type approach is the, how do I get in a position to get ahead? Yep. And if your, if your paradigm is, I'm going to deploy this in production to 10 systems, then a hundred systems, then a thousand systems. And then we're going to stop and evaluate, is this the right technology for us? And instead of buying it for the entire environment, we're going to buy a thousand licenses, do a pilot deployment and see if it does what we want. That would be a superior strategy for getting to full production deployment, but we just don't do that. Because we're not really equipped for that sort of flexible deployment approach. Okay. So just, it's an observation that, that I have for this. And I think it's a, --- Page 11 --- I think it's an interesting one. Another thing that I, that I tried to do, and I'm going to show you some of this code because it's so crazy what the results came out with initially. I basically said, Hey, Gemini, I want you to take these five documents that I have and I want you to do a year over year comparison. And I'm going to show this to you when I did it. And it did it. And it was showing me all these things and I was like, that's cool. That's weird. Why is that? And I, I'm like, I'm looking at it. And I'm like, what? And I started taking outside of what it was doing. Can you, I don't know if this is crazy sounding to you, but can you imagine that it was taking questions that had nothing to do with one another and doing correlations year over year, that's exactly what it was doing. Okay. I'm going to show you that. So what I needed to do to fix it was I needed to go in and basically manually reference all of the questions, do my cross-referencing. I used Gemini to help me to do it, but it basically was like I couldn't just say, here are five spreadsheets, figure out what the things were that were cross-referenced. And it took me tens of hours to actually get this to where it was even working. Now, it's a big, complex survey. There are hundreds of questions in each one, and there are minor changes that are there. I thought it was going to do a much better job than it actually did. So this is the sort of stuff that I started coming up with. This is my SOC architecture. Do you have a single centralized one, or do you have a distributed one? There are different choices on this. Again, the details of the results don't really matter at this point. What's interesting to me is the methodology that enabled me to do year-over-year comparisons. Because I'm doing it for a specific thing, I hope you get some ideas about what you could do in your environment so that when you go back and start trying to articulate to your management what the SOC is doing well and what it's not doing well, you can show some of these longer-term trending things. And I've got a ton of code that you can take as an idea for this. And this is the relationship. I already showed you this one. I'll show you how I did this. So I'm not going to run the SOC survey with SANS in 2027. We'll do something else with SANS. It's something that I haven't, with the analyst program, exactly decided what that's going to be. It's been 10 years. I've been doing this for a while. I'm just not going to do that. What I am going to do, though, probably next year, to be able to do something related to this. --- Page 12 --- As I've noticed over the last... Over the last two years, there are a lot of other SOC surveys that vendors are sponsoring. I've participated in the SOC CMN one for the last couple of years with Rob Venos, a partnership with SOC CMN, and I do that. And then I've noticed all these other vendors are actually doing stuff. So next year, what I think I'm going to do is I'm going to take all of the 2026 surveys that other vendors published, and I'm going to try to correlate that information and see if they're telling the same story or not. That seems interesting to me. So that's probably what I'm going to do next year for the survey. It's going to be more of like a survey of American literature, a survey of cybersecurity reports kind of thing, like survey across all these different things that are being published, because that's interesting to me. So just a couple of things for links. If you want to message me on LinkedIn, there's that. I have a ton of presentations of things that I share. You can go to this montance.com prez. And then the SOC survey links to all of the years that are there, downloadable from the SANS website, or the reference to all the Jupyter Labs that I've released, those are all there. So I'll give a second for this, and then I'm going to go to the Jupyter Lab and just read you some of the stuff that's in there. Now, this starts to get kind of complicated, so I'm going to do my best to keep it simple. But let's go into this. So here in this, I've already launched the Anaconda, and it's running, and I've got the environment, and I started Jupyter Lab, and I already ran the whole thing, so it's good to go. This one is a raw cell. If you don't have the dependencies, you could switch this to code and then run it, and it'll install those things. I don't need to do that, because I've already installed it in the environment, but that's a way that you get it set up. Then I have this. You know, the full deident Excel spreadsheet of the stuff that's there. I do some double checks. This is what the survey looks like. Here we've got this text of the things that are in place. I have my original column titles that are inside of there, so these are the details that are there. I then do this thing to build some output, so I can actually relabel my columns in my dictionary. Skip ahead. You don't need to see all of that. So this is where I do my work to make sure that all the questions make sense, that everything's aligned and in place, and yeah, I manually go through all of those and double check it. And then down here, I've got different variations on it, because I've got different years and --- Page 13 --- things that I'm validating in this. So down here, then I end up with this is the asset. Rename 2026, because I have different things in different years, different columns, name different stuff. I have a per year basis for this, and then I just split it into two different sections. So here's my capabilities section, right? So all these different things around capabilities that are expressed. Then when we get into part two, we have stuff around technology and budget and other stuff that's in there. Okay. So now I just sort of. Take a look at this, make sure that this is good to go. I look at one column, I then do an in name replace on the columns. So now I have my, my column names that are what they need to be. Okay. So that's great. So then I end up with this data frame. If you're not familiar with pandas, that's that's fine. So here I have this mean minutes to complete 102, which is pretty high on average. It took people an hour and 40 minutes is what this says. Which is a little bit weird. So then I look at this and I actually trim this out and the trimmed mean, I have 47 minutes, which I showed you that already. And then this is my, my histogram of elapsed time. So most of the people are in are in this category. This one person is in this other category and it completely skewed the mean. So that's the justification of dropping it where people were this year, we actually did the survey in Spanish. Portuguese and, uh, and English. I've been trying for years to, uh, to expand languages. I did it in Japanese one year. I tried to get it to run an Arabic translations, a problem. But what I've found is like, if I can give a survey in a language, that's a native language, I get a different composition of people responding, you know? And as an example, it's sort of like how many, uh, how many socks are there in India? A lot of socks in India. And I don't have any insight to the people who are working in those socks. I can't get the information from them. They're not sharing it because they're not going to take this survey, but if I could offer it in one of the Indian languages, I could probably get more responses. And I've, I've done this year over year, uh, to try to expand it. And it's never, it's never really stuck. Just something I've tried and tried and tried. So here we have different countries that are there. Um, and this is, uh, this is how many countries that were there. Just stuff. Walking. Going through this. These, uh, aren't especially interesting. I'm going to skip ahead for this. I started, um, doing this years ago where sometimes I have a question where it's like, okay, I have five answers. --- Page 14 --- I can just give a chart on that. But sometimes I have 30 different answers. It's hard to have a chart on something where there are 30 different answers. So I have this stuff where I label things as difficult and I start going through this. I need to skip ahead on this one because all of the, uh, visualizations. I ended up doing these visualizations for all of that. And then reviewing the, uh, the output. So here is an example. This is the, uh, this is the data visualization of every single question inside of here. I also build myself things like, um, word clouds. So on the text questions, I just auto build a word cloud so I can get a quick sense and the word cloud. It basically is like the, the larger font. Pitch. Pitch is the more commonly occurring word. So I just do this for myself, uh, to have it very fast, uh, to see, see what's inside of their, um, stuff related to metrics. So I go through and pull out the metrics and the details for that. That's not especially interesting at the moment here. Now I have my different metric visualizations and looking at the stuff of what's there. Yeah. And that's, uh, I do this a bunch of different ways. Okay. Um, give myself different pictures, top five of stuff counts and so on. Skip ahead. Okay. Here's the technology stuff. Technology is always fascinating to me because we're such a, um, we're such a tech focused, um, sort of group. So this is my satisfaction. How satisfied are you with these? Um, and you know, the different categories of technology that's, uh, that's present. And I kind of slice this a bunch of different ways to see the, uh, to see the results. Uh, that were there next-gen firewall. This was, this was in the, uh, the visual as well. Um, and I showed you that code already. I like this, uh, this compound view, um, for, for this, what else do they do? Okay. No, this is all still tech. That's boring. Oh, this one, this is a thing that I actually, um, like seeing for myself. This is my deployment state. Production. All production, partial implementing per purchased or planned, um, ranked on, uh, on GPA. So I blend the two to give myself a visual in terms of, uh, in terms of what this looks like. My ranking. No, I must be ranking. I'm I'm ranking here. I'm ordering on production. All is where I'm ordering that. So anyway, just more things sort of like, oh, what does this look like? And then how can I cross-reference that to other data? Uh, that's in place. So that's useful. And then this is me just playing around with the visualizations. It's not anything new inside of there, all the same stuff, trying to make it, trying to make it look, you know, nice. It just gets really confusing. --- Page 15 --- I saw somebody, when I put that first one up, they're like, whoa, what am I looking at there? Right? So this is the, uh, this is the one that I showed you the status count, uh, deployment and GPA. Um, so this is there. All right. Different stuff on technology. I'm just going to skip ahead because what I really want to get to in the last few minutes here is this, um, year over year stuff. All right. And then I, this is where I was doing some of the cohort stuff, this person versus this, this group versus that group. Are they generally more satisfied or are they generally more capable? All these? Uh, all these different things that I cross-referenced and the cohorts, I mentioned that as well. So a bunch of different cohort analysis. Oh, then this is where I start getting into the, um, statistical analysis. So I basically said, go through, take the items, um, inside of here and start doing a statistical sweep to see where I have correlations among the data, because a correlation indicates that like, maybe I could use one thing to have a comprehension. Of another thing. So I started doing these, uh, you know, these different tests, um, statistical significance related to, uh, related to this. Um, and this is all still in that cohort approach that I took to try to say, you know, how are, how are these T these teams in their approach different? Um, so here, then I had some, again, more statistical analysis, lots of details in terms of, uh, you know, working through this to try to find something statistically significant. All right. Comparison for delivery models, and I'm just doing more AB testing. Oh, what else do I have here? What's this one images capabilities. This is, and I, you know, I save all this stuff out in case I want to use the visualizations for reporting, okay. Year over year analysis. So this is where I started doing the year over year analysis. Okay. And this is my note to you, a bunch of these are garbage, but the reason why I leave the garbage in is because I don't just come up with the, like the analysis code off the top of my head, I can't just write it. I try and fail over and over and over again to see what's going on. Okay. So what I do for code like this one is I switch it to raw, like that didn't work. Like it was a starting point for the stuff, but I basically did this. So now I have these different functions where I'm mapping these different years. So there's 2020, 21, you know, and so on. And I'm basically then doing this this comparative analysis between the years. So here's one where I basically do the 20 and 22 for, for this, this is how I rename the specific columns. Cause I have this like consistent column expression. --- Page 16 --- Okay. I know. I know. Then what I do here is I have you know, to, I have all of the old data. I pull it in, in order to be able to do this comparative analysis and I'm doing my my crosswalk. These are the different questions. This is how many years the question was present and whether there was stuff that sort of changed across him. And this is where I was trying to use Gemini to help me. And it really just did not, did not work. Very well. And I ended up going through some of some of this stuff manually after the fact, let me show you one of these where it's like, I'm having it originally, I'm having it say, okay, just build me the year over year trend to go find the thing for this you know, for this question and tell me the details of it. And then as I was looking through this, I was like, Hmm, some of these don't make sense. So what I did eventually. Was I said, okay, um, what you need to do, and I'm going to switch ahead to this from a, from an analysis perspective, because this is me finding like, oh, this thing has lost its lost its mind. I can't trust it to, to, um, to, to find the correlations. What I started doing was saying, okay, give me the questions that you're using in, in each survey to be able to actually show me how you're doing this correlation. Okay. Cause I was having it write the code to be able to do this correlation stuff. There's one that I want to show you where this is, this is what, uh, what came out where it's like, these are the questions that it found that it thought were the same. And they in fact were not, uh, not yet. Sorry. I'm going to scroll down fast. Don't look at the screen. It's going to be blinking lights, this stuff. There we go. So here I was like, okay, hold on a second. How are you doing this correlation stuff? Show me what you're doing. So original question was like, please indicate which technologies, you know? And then it was saying, okay, well I found this yes answer. And that's what they're. And in 2026, I used rate your satisfaction with the technology. It's like, these are completely different questions. Okay. And then the other thing, the data, I said, do that, write the code to correlate based on this. And it's, and it decided that these were the data points that it needed to correlate. Yes for this one planned for these and see for that one. And I was like, okay, need to start that over completely. So what I ended up needing to do, and this, like, if you just look at these, it's like, you know, catastrophe. Yeah. It's going to be after catastrophe for each of them in terms of the correlation that it --- Page 17 --- came up with. Now I'm telling you, I'm spending time just like, what in the hell is this thing? I wanted it just to find trend, you know, give me, give me the trend information, but it, it didn't do a very good job of that. So then what I ended up with is me writing the, the mapping, I ended up having to say, these are the things that you can look for. These are the correlated points. And these are the details that I can work on. And I actually had to dramatically reduce the scope of the year, year, year over year trending because I just couldn't write all of it. And the, and the, the GPT couldn't help me to do that. So here I was saying, these are the things that are there. Look for these changes. I had these different, you know, things. Yeah. Yeah. Yeah. These are the logical things that were in place. I also had specific mappings that were, that were in place for this. Okay. And here, again, this just, this just went, went through and failed and failed and failed for it to have logic to, to help on it. So eventually I had to go and write my map. So here's what I did. I basically had a, a mapping where I ended up writing out, okay, this is what I'm going to go from this thing to this thing to be able to actually do it. Okay. And then after I wrote the mapping, I was able to start to visualize the, you know, the items that were present, right? So this was just iteration after iteration. And then after iteration to get this to where it was actually working in the trending of what was in place, man, once I got those things working, I picked some stuff that was useful. So I'm gonna skip to the bottom and come back to this. And these are some of the visualizations that ended up working for them. Okay. After I, after I specifically mapped them to say like, this is, this is what needs to, this is what needs to be expressed. Okay. So I basically changed my approach saying, scan the spreadsheet, look for this specific thing, pull these out, show me the values, show me the expression. Now that I've actually seen the data values for each one. Now I'm comfortable actually doing the year over year comparative analysis for those. Okay. And so that's what I ended up doing for this. And man, I thought I was going to have all these amazing year over year things. And 20 hours later, I was like, I got two. Okay. Okay. But, but if now I want to go and do another year over year mapping, I figured out how to do it. I was like, ah, that's good enough for my purpose. Cause it's just like, this is, this is the thing I kind of solved the problem of how --- Page 18 --- to do the year over year mapping and correlation stuff. So, you know, this is just like a man, me sharing a bunch of code of stuff that I worked on for a couple of reasons. Okay. One is, um, I know nobody's going to do this, but somebody please check my work. If you can understand and follow it, like it is insanely complex. And this year I actually use Gemini to help me write a lot of this code. Okay. Based on things that I did previously, I just extended it substantially. And I was able to make a lot of, a lot of additional analysis. The other thing is take it and answer your own question. I don't know what your question is, but you answer it for yourself. Try one thing, look at the code. If, if you actually are trying to do that and you have questions, you can message me. Okay. Now there are five minutes left. Wait right here. Talk amongst yourselves. I'm not kidding. Wait. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. you you you you you you you all right all right so as a thank you for uh for attending and uh to hopefully get you convinced to make metrics that don't suck this is this is a book that i wrote recently this is a book that i wrote recently um on the value of security operations um so i've got more hold on there we go um this is something that the book is really short no surprise there's a downloadable companion jupiter lab notebook that exists for you to uh to be able to actually do the to do the calculations other do you have one of these so um and the the idea behind it is is basically once you figure out how to plug some of your data in all of the rest of the visualizations just cascade the visualizations at the end of each chapter are there as metrics that i would want you to be reporting and having discussions around it and the code is there you download it but it basically makes those visualizations in the same way that i've done this here and this is intended to help you to talk to your organization to start to have that conversation around what does better really look like it's it's not about how many tools do i get to production it's about the tools making a difference that actually drive things that are meaningful for your organization okay so with that i'll also say for folks uh for folks online since you uh since you don't have uh the the physical book i'll say if you um if you email me at this uh at this email um i'll send you a uh a code for the e-book --- Page 19 --- um for for those of you online um if you didn't if you want it uh so that's available just send me a message and i will uh i'll send you a code back until you can download the e-book okay so appreciate your time and attention i hope that you um i hope that you do good stuff with this i hope that it was insightful in some way for you and uh hope the rest of the week goes well for you for your class all right you bet see you