Denver Equifax SO
Presented by: Christopher Crowley
PresentationThe Denver Equifax SO presentation focuses on security operations and avoiding being the next Equifax, with a core emphasis on the Analysis of Competing Hypotheses (ACH) methodology. Key concepts include the importance of decomposing complex problems, externalizing mental machinery biases, and using structured analysis to identify the most likely explanations for security incidents.
Full Transcript / Content
--- Page 1 ---
2019-07
Security Operations to
Avoid Being the Next Equifax
Christopher Crowley - CCrowMontance
--- Page 2 ---
Copyright Christopher Crowley
Security Operations
2
Introduction
--- Page 3 ---
Copyright Christopher Crowley
Security Operations
3
Christopher Crowley
- Background: Had root on most systems in employer
at 15 years old (Not much #CYBER in the 80s)
- Sectors: Defense, Education, Energy, Government,
Financial, Software Development, Telecom
- Regions: US, Europe, Middle East, Asia, Australia
- Currently: Consultant, author of (SANS deprecated)
MGT517. Which is now: SecOps (soc-class.com),
SANS: SEC511, SEC575, SEC504, …
- SOC build timeline project:
https://www.montance.com/soc/timeline
SANS Senior Instructor
Twitter: CCrowMontance
--- Page 4 ---
Copyright Christopher Crowley
Security Operations
4
Presentation Available to Download
https://mgt517.com/soc
You can take photos & tweet if that’s
your thing, but I would really like you to
pay attention to me, which is why I’m
making this slide deck available to you
Stop taking notes
--- Page 5 ---
Copyright Christopher Crowley
Security Operations
5
Terminology & Background
--- Page 6 ---
Copyright Christopher Crowley
Security Operations
6
How This Will Work
- I’m going to give you some background so
you understand where I’m coming from
- This is a lot of terms, technology, mindset,
etc.
- It will be helpful when I get into the story
that I want to tell you
My Approach
--- Page 7 ---
Copyright Christopher Crowley
Security Operations
7
Full Spectrum Capability
- The US DOD Defense
Science Board describes
adversaries in six tiers
- The top tier attackers are
nations state funded, and
are in the “Billion Dollar
Club” in capability of
spending
- This type of attacker can
bring the full capability of
traditional intelligence
capability
Threat Taxonomy
--- Page 8 ---
Copyright Christopher Crowley
Security Operations
8
Examples
- MITRE / Carson Zimmerman: https://mgt517.com/csoc-mitre
CSOC – Cyber Security Operations Center
--- Page 9 ---
Copyright Christopher Crowley
Security Operations
9
Functional Areas of a SOC Crowley Defined
Steering
Committee
Command Center
Network Security
Monitoring
Threat Intelligence
Incident Response
Forensics
Self-Assessment
--- Page 10 ---
Copyright Christopher Crowley
Security Operations
10
Models of Analysis Weltanschauung
- Caltagirone, Pendergrast, Betz
- The Diamond Model of
Intrusion Analysis
- All events have the elements:
adversary, infrastructure, capability,
and victim
- Edge connected, sequenced graphs of these
nodes are Activity threads
- Events are described by tuples with
feature and confidence E = {f1,c1; f2,c2;…}
Diamond Model
--- Page 11 ---
Copyright Christopher Crowley
Security Operations
11
Models of Analysis Weltanschauung
- Hutchins, Cloppert, Amin published
Intelligence-Driven Computer Network Defense Informed by
Analysis of Adversary Campaigns and Intrusion Kill Chains
- Based on military targeting doctrine of
- Find, Fix, Track, Target, Engage, and Assess
- Intrusion Kill Chain is: Reconnaissance, Weaponization,
Delivery, Exploitation, Installation, Command and Control,
Actions on Objectives
Phases In Diamond Model: Kill Chain
--- Page 12 ---
Copyright Christopher Crowley
Security Operations
12
Diamond Model What Else?
Interruption: Beyond Prevent/Detect
Adversary Interruption Explanation
Detect Enhanced methods for identifying the presence of
adversary activity.
Degrade Reduce the ability of the adversary to perform operations,
but not able to completely stop actions.
Disrupt Stop adversary action while it is occurring.
Deny Prevent adversary from completing an action.
Deceive Provide misinformation to adversary to cause expenditure
of resources, cause mistakes, and/or facilitate detection.
--- Page 13 ---
Copyright Christopher Crowley
Security Operations
13
Phin Phisher Defense Hack Back - A DIY Guide
Phineas Phisher vs. Hacking Team – Diamond Model - Initial Entry
Kill Chain Phases External Facing Appliance Notes Interruption?
Reconnaissance A: Phin; I: compromised assets; C:
network scans; V: HT network
Scanning
performed
Weaponization A: Phin; I: unknown; C: exploit
development; V: appliance OS
Develops custom
exploit
Delivery uncertain
Exploitation uncertain Gather more info?
Installation A; I: custom firmware;C: firmware
;V: appliance
Custom firmware
Command and
Control
A; I: DNS servers; C: remote
access; V: appliance
DNS Control
Channels
Action on
Objectives
A; I: Network access in HT DMZ;
C: firmware implant; V: appliance
Initial entry grants
access to
environment
--- Page 14 ---
Copyright Christopher Crowley
Security Operations
14
Cultural Dimensions Weltanschauung
- Power Distance Index (PDI)
- Individualism versus Collectivism (IDV)
- Masculinity versus Femininity (MAS)
- Uncertainty Avoidance Index (UAI)
- Long Term Orientation versus Short Term Normative
Orientation (LTO)
- Indulgence versus Restraint (IND)
Geert Hofstede – National Culture
--- Page 15 ---
Copyright Christopher Crowley
Security Operations
15
Cultural Dimensions Weltanschauung
- Means-oriented vs. Goal-oriented
- Internally driven vs. Externally driven
- Easygoing work discipline vs. Strict work discipline
- Local vs. Professional
- Open system vs. Closed system
- Employee-oriented vs. Work-oriented
- Degree of acceptance of leadership style
- Degree of identification with your organization
Geert Hofstede – Organizational Culture
--- Page 16 ---
Copyright Christopher Crowley
Security Operations
16
Sequence of Equifax Events
--- Page 17 ---
Copyright Christopher Crowley
Security Operations
17
Walk Through
- Next we’ll talk through what is known of the
sequence of events
- I’ll make important points as we go
- But after the full sequence we’ll review
lessons learned in detail
- This is all in 2017 (except the SSL certificate)
Will Discuss Lessons Learned Afterward
--- Page 18 ---
Copyright Christopher Crowley
Security Operations
18
Sequence of Events Equifax, 2017
Jan
31,
2016
Certificate allowing SSL visibility expires
Feb
14
Struts vulnerability identified
Mar 7
Struts Public Disclosure - CVE-2017-5638 –
CVSS BASE: 10.0
--- Page 19 ---
Copyright Christopher Crowley
Security Operations
19
Sequence of Events Equifax, 2017
Mar 8
Equifax notified by US-CERT (Probably
generic notice to many orgs)
Mar 9
Internal Equifax notification: Patch Struts
within 48 hours
Mar
10
Post exploit discovery reveals this is initial
entry date / time by intruder (48 hours)
--- Page 20 ---
Copyright Christopher Crowley
Security Operations
20
Sequence of Events Equifax, 2017
Mar
14
Internal Emerging Threats team releases Snort
rule to identify attack traffic.
Countermeasures team installs it
Mar
15
Updated McAfee signatures provides check for
Struts vulnerability. Scans conducted, no
vulnerabilities reported
Mar
16
Global Vulnerability Team briefs on
vulnerability and tells teams to patch,
everyone sent copies of slides
--- Page 21 ---
Copyright Christopher Crowley
Security Operations
21
Sequence of Events Equifax, 2017
May
13
Struts front end to ACIS system – ACIS houses
customer information, and hasn’t been
updated in decades
May
13 –
July
30
Attackers gain initial access via Struts,
maintain access using installed webshells
Infra
Two application servers, two web servers
Firewall protecting network segment, but web
servers exposed
--- Page 22 ---
Copyright Christopher Crowley
Security Operations
22
Infrastructure Diagram (Simplified)
--- Page 23 ---
Copyright Christopher Crowley
Security Operations
23
Sequence of Events Equifax, 2017
May 13 -
July 30
Web shells installed on both application
servers from web servers
May 13
– July
30
Accessible file share mounted, contained
file with plain text credentials (contrary to
policy)
File shares not segmented within “legacy
environment”
--- Page 24 ---
Copyright Christopher Crowley
Security Operations
24
Sequence of Events Equifax, 2017
May 13
– July
30
Credentials stolen from file shares used to access
database servers
Segmentation of databases not performed, all
Equifax DBs available from within ACIS
May 13
– July
30
Access to the data base servers included queries for
meta data to identify the information found
within tables
9,000 total queries performed, undetected and
unchallenged
--- Page 25 ---
Copyright Christopher Crowley
Security Operations
25
Sequence of Events Equifax, 2017
May 13
– July
30
9,000 queries, 265 had PII information
No secondary protection such as encryption
within the database for PII data
Infra
Visibility for monitoring was expected to be
performed by SSL intercept capable
inspection of traffic into the ACIS network
--- Page 26 ---
Copyright Christopher Crowley
Security Operations
26
Sequence of Events Equifax, 2017
(January 31,
2016)
Certificate allowing SSL visibility
expires
--- Page 27 ---
Copyright Christopher Crowley
Security Operations
27
Sequence of Events Equifax, 2017
July
29
Countermeasures team installs new certificates
to SSLV system (67 new certs)
July
29
Monitoring occurs, due to newly returned
visibility
July
29
Obvious unauthorized network connection
from ACIS environment to IP address in
China
--- Page 28 ---
Copyright Christopher Crowley
Security Operations
28
Sequence of Events Equifax, 2017
July
29
Full PCAP tool used to inspect traffic related to
specific IP address
July
29
10 MB of data transferred during this session
No previous full PCAP available due to SSL
cert
July
29
ISP for this IP address blocked
--- Page 29 ---
Copyright Christopher Crowley
Security Operations
29
Sequence of Events Equifax, 2017
July
30
Vulnerability Scanning of ACIS environment
performed
July
30
Vulnerability scan results indicate SQL
injection flaws, and insecure direct object
reference issues
July
30
Forensic investigation of hosts indicates that
PII was access by unauthorized resources
--- Page 30 ---
Copyright Christopher Crowley
Security Operations
30
Sequence of Events Equifax, 2017
July
30
Second malicious IP address – in Germany
identified as accessing ACIS environment
12:41pm
ACIS environment taken offline “for
emergency maintenance.”
July
30
Executives and Senior Executives briefed on
status of breach
--- Page 31 ---
Copyright Christopher Crowley
Security Operations
31
Sequence of Events Equifax, 2017
~Aug 14
Effort of 50-60 people start working on
remediation effort (Project Sparta)
Substantial effort put forth to develop
customer facing portal
Sept 4
143 Million affected users identified
--- Page 32 ---
Copyright Christopher Crowley
Security Operations
32
Sequence of Events Equifax, 2017
Sept 7
Public notification
Website, Call Centers
Formal notification to all 50 states
--- Page 33 ---
Copyright Christopher Crowley
Security Operations
33
Sequence of Events Equifax, 2017
Sept 7 - 30
Website clones generated
Official Equifax twitter tweets link
to wrong site
Mis-information provided to
customers
Website intermittently unavailable,
code inefficiencies due to short
fuse development
--- Page 34 ---
Copyright Christopher Crowley
Security Operations
34
Sequence of Events Equifax, 2017
Sept
15
CISO and CSO retire
Sept
26
CEO Retires
Oct 2
Forensic investigation finalized, number
increases from 143 to 145.5
CIO for Global Platforms (ACIS) fired
--- Page 35 ---
Copyright Christopher Crowley
Security Operations
35
Sequence of Events Equifax, 2017
Early 2018
Affected individuals
calculation rises to
~148 million
--- Page 36 ---
Copyright Christopher Crowley
Security Operations
36
Sequence of Events Equifax, 2017
Final Details of Data Breach
- Final Stats
- Name: 146.6M
- DOB: 146.6M
- SSN: 145.5M
g@M3 ()\/3R
--- Page 37 ---
Copyright Christopher Crowley
Security Operations
37
Lessons Learned from Equifax
--- Page 38 ---
Copyright Christopher Crowley
Security Operations
38
Lessons Learned
- When there are lots of items like this, I try to
enumerate them in an ordered fashion, so I’m going
to go through the timeline above and point out some
items. Would take days to consider and report all
- I typically would do this with Kill Chain or Diamond
Model to explain the intervention opportunities,
next slide is a repeat example from another famous
compromise: Phin Phisher attacking Hacking Team
Where to Start?
--- Page 39 ---
Copyright Christopher Crowley
Security Operations
39
Phin Phisher Defense Hack Back - A DIY Guide
Phineas Phisher vs. Hacking Team – Diamond Model - Initial Entry
Kill Chain Phases External Facing Appliance Notes Interruption?
Reconnaissance A: Phin; I: compromised assets;
C: network scans; V: HT network
Scanning
performed
Weaponization A: Phin; I: unknown; C: exploit
development; V: appliance OS
Develops custom
exploit
Delivery uncertain
Exploitation uncertain Gather more info?
Installation A; I: custom firmware;C: firmware
;V: appliance
Custom firmware
Command and
Control
A; I: DNS servers; C: remote
access; V: appliance
DNS Control
Channels
Action on
Objectives
A; I: Network access in HT DMZ;
C: firmware implant; V: appliance
Initial entry grants
access to
environment
--- Page 40 ---
Copyright Christopher Crowley
Security Operations
40
Lessons Learned
- A good way to reinforce a lesson is to review it shortly after
the lesson, to reinforce the thoughts
- If you want, take this slide deck, go through the preceding
Equifax sequence and enumerate suggested interruption
opportunities
- In the interest of time, I’m going to discuss some of them
in the following slides
- Feel free to send your results to me:
chris @ montance.com
Your Homework
--- Page 41 ---
Copyright Christopher Crowley
Security Operations
41
Reminder: Diamond Model What Else?
Interruption: Beyond Prevent/Detect
Adversary Interruption Explanation
Detect Enhanced methods for identifying the presence of
adversary activity.
Degrade Reduce the ability of the adversary to perform operations,
but not able to completely stop actions.
Disrupt Stop adversary action while it is occurring.
Deny Prevent adversary from completing an action.
Deceive Provide misinformation to adversary to cause expenditure
of resources, cause mistakes, and/or facilitate detection.
--- Page 42 ---
Copyright Christopher Crowley
Security Operations
42
(January 31,
2016)
Certificate allowing SSL visibility
expires
Interruption
Opportunities
Sequence of Events Equifax, 2017
Detect:
Scan for
certificates
Detect:
SIEM feed
health
Disrupt:
Web app
firewall
Deny:
Fail Closed
--- Page 43 ---
Copyright Christopher Crowley
Security Operations
43
Zimmerman: Coverage
- SIEM report assessing the feed performance
- Would have
easily
identified the
SSL visibility
lapse
Coverage Report Example
--- Page 44 ---
Copyright Christopher Crowley
Security Operations
44
Crowley Incident Avoidability
- On a (discrete) scale of 1,2,3, was it avoidable?
- 1 means a measure, already available in the
environment wasn’t applied and resulted
in the incident
- 2 means a measure is available in the larger
environment and something (economic,
political) prevents implementing it within the
organization
- 3 means nothing is available to prevent that
method of attack
- Assess every issue identified in this way
Simple: 1, 2, 3 1
- Patch available, but not applied
2
- Patch available,
risk decision
not to apply
3
- No patch
available (zero
day)
--- Page 45 ---
Copyright Christopher Crowley
Security Operations
45
Crowley Incident Avoidability
Equifax Score: 1
1
- Patch
available,
but not
applied
--- Page 46 ---
Copyright Christopher Crowley
Security Operations
46
Crowley Incident Avoidability
- You don’t care about US Congress (or Equifax Board
of Directors), but when you have to go explain what
happened to whatever authority you answer to, do
you want to be explaining why you were a 1?
- They were all CYA, fired this person, we’re making
improvements, etc.
Equifax Score: 1 1
- Patch
available,
but not
applied
--- Page 47 ---
Copyright Christopher Crowley
Security Operations
47
Impact Level Definition
- Few systems (or only a specific type)
- Unimportant systems
- Unimportant dataLow
- More systems (or many common types)
- Important or high value person’s, account, or system
- Important data at riskModerate
- Most systems (or almost all types)
- Highest level accounts, users, and systems
- Business critical dataHigh
Does Your Management Understand This Already?
--- Page 48 ---
Copyright Christopher Crowley
Security Operations
48
Impact Category Definition
- Low – minimal function disruption
- Moderate – substantial disruption
- High – complete disruption
Functional
- Intellectual Property (L/M/H)
- Integrity Manipulation (L/M/H)
- Privacy violated (such as PII / PHI)
Informational
- Regular – predictable using resources on hand
- Supplemented – predictable with augmented resources
- Unrecoverable – data breach which cannot be undone
Recoverable
Does Your Management Understand This Already?
--- Page 49 ---
Copyright Christopher Crowley
Security Operations
49
Scale: 1-10
3 7 10
Low Moderate High
3 Functional 9 21 30
7 Informational 21 49 70
10 Recoverable 30 70 100
Impact Quantification
Example Quantification
Current Incident 191
Legacy System #8 - ACIS
--- Page 50 ---
Copyright Christopher Crowley
Security Operations
50
More Metrics?
- Carson Zimmerman and I co-authored a talk that he
gave at FIRST last month:
- https://mgt517.com/first-metrics
If You Like Metrics
--- Page 51 ---
Copyright Christopher Crowley
Security Operations
51
The Key to This Talk
- So, take the most important asset you have, and
think about how that isn’t being protected effectively
- Assume that protection will fail, then think about
how you can prevent catastrophic failure, as well as
what you’ll do when failure (small or large) happens
- Let’s check out how Equifax performed when they
identified the issue
Your Company, Like Equifax, Has Something of Value
--- Page 52 ---
Copyright Christopher Crowley
Security Operations
52
Communication Is Critical
Don’t Create a New Domain During a Security Incident
- First
version of
Equifax
imposter
page,
courtesy,
internet
archive
--- Page 53 ---
Copyright Christopher Crowley
Security Operations
53
Equifax Dumpster Fire
- Again, don’t
create a new
website
- If you do, be
sure you tweet
the correct
address:
What Not to Do During a Security Incident
https://www.equifaxsecurity2017.com/
53
--- Page 54 ---
Copyright Christopher Crowley
Security Operations
54
Resolution to Asset Discovery
- System recording the responsible party for each system
- Owner
- Contact for Operations
- Data experts
- Boundaries
- Scenarios of compromise (actual past, or industry specific)
- Risk management
- Incident handling specific details
- Public relations point of contact
- Legal requirements and notification
Fundamental Error – No Culpability/Responsibility: Until After Breach
--- Page 55 ---
Copyright Christopher Crowley
Security Operations
55
Does Anyone Do Asset Discovery?
- Seriously, is the technology
incapable of identifying
systems on your network?
- I think the disappointment is
mapping to useful
organizational contact
A Technology Problem?
--- Page 56 ---
Copyright Christopher Crowley
Security Operations
56
Matrix Depiction
Network
Infrastructure
- Owner: Dir
Network
- POC: Thomas
Racks
- Impact:
Moderate -
High
Server
Infrastructure
- Owner: CIO
- POC: Thomas
Racks
- Impact: Low -
High
Workstations
- Owner: Dir
Network
- POC: Thomas
Racks
- Impact:
Moderate –
High
Financial
- Owner: CFO
- POC: Sarah
Banks
- Impact:
Moderate -
High
JABB
- Owner: BB
Acct POC
- POC: Vincent
Lehman
- Impact:
Moderate -
High
AMD
- Owner: AMD
Acct POC
- POC:
Alexandra
Harris
- Impact:
Moderate –
High
GCCS
- Owner: GCCS
Acct POC
- POC: Victoria
Houston
- Impact:
Moderate –
High
MEH
- Owner: MEH
Acct POC
- POC: Iggy
Calendula
- Impact: Low -
Moderate
Generic Server
System
- Owner: CIO
- POC: SOC
Manager
- Impact: Low
– High
Generic
System
- Owner: CIO
- POC: SOC
Manager
- Impact: Low
– High
Generic Cloud
Hosted System
- Owner: CIO
- POC: SOC
Manager
- Impact: Low -
High
Part 1 - Capabilities Mapped to System Types
- Centralized knowledge base of known and unknown system types: reporting
requirements, service level objectives (SLO), POCs, Strategies, reporting SLAs,
customers, cyber insurance
- Handling strategies, guidance for impact determination, IP address ranges,
operating system types, exceptions, known vulnerabilities, previous incident
types, associated exercises, response capabilities applied based on per-system and
per-incident type
- Internal dependencies, cloud contracts, external auditors, legal council
--- Page 57 ---
Copyright Christopher Crowley
Security Operations
57
Matrix Depiction
Internal
Capability
Recon Weapon-
ization
Delivery Exploit Install Command
& Control
Action on
Objectives
Spam
Gateway + -- ++ +- +- -- -
EDR - -- + + + + +-
NIDS - -- + +- + + -
Network
Proxy + - ++ - +- ++ +
IR -
Containment - -- - - - + +
Part 2 – Defense Capabilities List Versus Kill Chain Phase
- List your capabilities against attacker cyber threat kill chain (or ATT&CK) phases, to
know what you have at your disposal in unfamiliar circumstances
--- Page 58 ---
Copyright Christopher Crowley
Security Operations
58
Matrix Depiction – Invincea Example
- Here Invincea has listed the NIST Cyber Security Framework (CSF) and identified
adversary TTPs (and specific targets) cross walked to capabilities and technology
--- Page 59 ---
Copyright Christopher Crowley
Security Operations
59
Equifax Issue: Sprawling IT
Product
Regional
Global
Global
Headquarters
APAC
Product 1 Product 2
Europe
Product 3 Product 4
In
fo
r
m
at
io
n
D
ir
ec
ti
o
n
--- Page 60 ---
Copyright Christopher Crowley
Security Operations
60
Response Capability Empowerment
- Does the police department ask permission to cordon off
an area, or drive cars fast to chase criminals? Yes, they
absolutely do. They have protocols, requirements, and
consideration for safety of innocent people.
- Identify the authority to conduct response on each type of
system, and what is to be done for unknown systems for
containment.
- Use business continuity plans for eradication and
remediation: don’t create new plans unless no business
continuity plan exists.
--- Page 61 ---
Copyright Christopher Crowley
Security Operations
61
Response Capability Empowerment
- Three characters of incident response
- Janitorial Services
- Fire Fighters
- Apex Predators
- https://www.mgt517.com/ir3
--- Page 62 ---
Copyright Christopher Crowley
Security Operations
62
Technology Selection Issues
- Inadequate “glue” to connect the different tools
- Poor selection of tools for the given purpose
- Often a result of buying tools prior to understanding the
intended purposes
- Protection scenarios expressed as “use cases”
- Video of Crowley’s presentation on use case development
from 2017 SOC Brief is available:
https://mgt517.com/use-case (registration required)
--- Page 63 ---
Copyright Christopher Crowley
Security Operations
63
Technology Selection Issues
- Whatever technologies are selected, each
changes with some frequency and must be kept
up to date
- SIEM in particular, but all other tools require
dedicated staff to keep it tuned and operating
effectively. Dedicate time for this
- This is sometimes transferred to a managed
service provider (MSP/MSSP)
--- Page 64 ---
Copyright Christopher Crowley
Security Operations
64
Technology Maintenance Issues
- SOC relation to standard IT?
- Oversight
- Embedded
- Coordinating
- Figure out what you’re going for
- Sequence of growth/maturity is often:
- Embedded (security activity in name only)
- Coordination / Oversight (separate to grow)
- Embedded (mature organization with clear
picture of what is needed and optimizes effort)
Oversight
Coordination
Embedded
--- Page 65 ---
Copyright Christopher Crowley
Security Operations
65
Resolutions: Security Staff
Empowerment & Reality Check
--- Page 66 ---
Copyright Christopher Crowley
Security Operations
66
Training, Exercise, and Practice
- There’s a large group of people responsible
that allowed that TLS certificate to expire
- It’s some mix of cultural laziness,
powerlessness, pride, meekness, etc. but
they missed it for a long, long time
- Security failed at basic IT Operations
Someone, Some Group, Some Manager, Some Executive…
--- Page 67 ---
Copyright Christopher Crowley
Security Operations
67
Training, Exercise, and Practice
- Seek funding for external training
- Cultivate internal training development
- Practice tasks – required self-improvement time
- Scripts, Internal tools, Coding, Mess remediation
– 2 hours per week, required, for example
- Encourage external presentation by paying for travel to
conferences if staff are approved for presentations
- Includes time for threat hunting, and other non-alert
driven work
--- Page 68 ---
Copyright Christopher Crowley
Security Operations
68
Training, Exercise, and Practice
- You should develop training &
certification plans per staff role
- Each role has certs that are
- Required for hire (or completion
within x days of hire)
- Standard training plan sequence
- Optional to training plan (but
company will pay for)
Simple or Complex Training Plan
--- Page 69 ---
Copyright Christopher Crowley
Security Operations
69
Monthly Staff Presentations
- SOC staff rotates on presenting current
technique, issue or problem
- This can be an unresolved problem, seeking
input and assistance from all SOC areas
- Opportunity to share methodology, thought
processes, and tools
- Should be a formal presentation (slides, Q+A)
Self-Improvement, Group Format
--- Page 70 ---
Copyright Christopher Crowley
Security Operations
70
Quarterly Internal Training
- External training is limited due to lack of time
and/or funding
- Resolution: Develop internal exercises
- Quarterly release of new exercise
- Based on current issues and events
- Additionally, results in exercises useful for:
interviews, onboarding, and keeping staff sharp
Group Self-Challenge
--- Page 71 ---
Copyright Christopher Crowley
Security Operations
71
Analytical Methodology
- Almost no organizations that I encounter have a
methodology in place for performing analysis
- There should be a training on analysis as well – few
places have this
- SANS SEC511 has some of it, ChrisSanders.org has
an analysis class online
- I’ll discuss Analysis of Competing of Hypothesis
(ACH) next
Group Methodology for Collaborative Performance
--- Page 72 ---
Copyright Christopher Crowley
Security Operations
72
Models of Analysis Background
- Human mental machinery is fraught with substantial weakness in
objectively processing information
- We overlook or ignore details that don’t suit our mental models
- We resist changing our mind once we’ve decided something is
right, even in the face of updated evidence
- Experimental results suggest that: experts in their field increase
self-assessed confidence level with more data, without
corresponding increases in accuracy of analysis
ACH – Mental Machinery
--- Page 73 ---
Copyright Christopher Crowley
Security Operations
73
Models of Analysis Background
- We use Situational Logic, Theory Application, and Comparison
- Leads to mistakes in analysis
- Make assumptions, but explicitly identify them in the analysis
- Common analytical failures:
- Satisficing – selecting the first, not the best
- Incrementalism – variation on same theme, insufficient breadth of scope
- Consensus – most popular, not the most accurate
- Reason by analogy – selection based on avoiding a previous mistake in analysis
- Principals – applying morality to select between the “good” and “bad”
ACH – Mental Machinery
--- Page 74 ---
Copyright Christopher Crowley
Security Operations
74
Analysis of Competing Hypotheses Background
- Decomposing a problem breaks it into
sub-elements to allow smaller units for analysis
- Externalization of a problem removes it from the
mental space it puts it in a tangible space to facilitate
more complex manipulation. Draw a graph, build a
model, use replicas of the affected systems and test
the scenarios you propose
ACH – Decomposition and Externalization
--- Page 75 ---
Copyright Christopher Crowley
Security Operations
75
Analysis of Competing Hypotheses Background
Incident Type Hypothesis 1 Hypothesis 2 Hypothesis 3
…
Data 1
Data 2
Data 3
Data 4
…
What’s Your Template for Analysis?
We’re going to develop these templates.
You’re not just to fill out the template, you still need
to think!
--- Page 76 ---
Copyright Christopher Crowley
Security Operations
76
Analysis of Competing Hypotheses ACH Applied
- Let’s look at the data we have, and list some potential
explanations
- Don’t criticize, kill, or otherwise refute any hypothesis! (We’ll
do that later.)
- This is hard for most people to do, since we tend to be
incremental
- Automation: populate hypotheses with all previously
encountered explanations for issues
1. Brainstorm
--- Page 77 ---
Copyright Christopher Crowley
Security Operations
77
Analysis of Competing Hypotheses ACH Applied
- Identify the data that we have
- Identify data that would be useful, and keep a
running list of it to try to acquire it
- Orchestrate: Based on the attempt to build out your
ACH templates, determine what data would be
needed for analysis, and work toward automatically
collecting this information through orchestration
2. List Data Attributes
--- Page 78 ---
Copyright Christopher Crowley
Security Operations
78
Analysis of Competing Hypotheses ACH Applied
- Make a chart with the hypotheses on the top and the
data elements (created in steps 1 and 2) on the side
- Don’t fill it in yet
- At some future time, this could potentially be
automated for an analyst given ranked / weighted
hypotheses sets based on data present for
investigation, (we’re probably not there yet)
3. Build Matrix
--- Page 79 ---
Copyright Christopher Crowley
Security Operations
79
Analysis of Competing Hypotheses ACH Applied
- Look for consolidation opportunities
- Are two hypotheses essentially the same?
- Can we consolidate data elements?
- Do we need to add more data elements?
- Don’t Automate or Orchestrate: Get the
human being to do expert work here
4. Assess and Refine Chart
--- Page 80 ---
Copyright Christopher Crowley
Security Operations
80
Analysis of Competing Hypotheses ACH Applied
- Fill in the numbers
- The method we’ll use is: 10 points per row total.
Assign the points for that specific data element (not
overall) that supports the hypothesis in each
column)
- Can’t exceed 10 points per row
- Sum the points at the bottom
6. Tentative Conclusions
--- Page 81 ---
Copyright Christopher Crowley
Security Operations
81
Analysis of Competing Hypotheses Background
Malware drive by download
attempt
Phone
actually
infected
Targeted not from site, but
from network user is on
Site Accessed when ad showed 5 2 3
Bad Grammar 3 3 3
Compulsion to risky action 3.5 3 3.5
Advertised only on one site 7 1 2
Threatening? 3.5 3 3.5
Reputation weight of domain? 4 3 3
Total: 26 15 18
ACH – Example – Spam Advertising
These values are the
analysis, different
analysts would have
different values
--- Page 82 ---
Copyright Christopher Crowley
Security Operations
82
Analysis of Competing Hypotheses ACH Applied
- Look at the matrix. Identify all the rows where the point
allocation is close (3,4,4, for example)
- Look at the matrix. Identify the rows where there’s a big
difference between points (10,0,0 or 7,2,1). These are the
data elements that your analysis hinges upon.
- Consider: what if that data was wrong? (Analysts dislike)
- This could be automated, but the benefit of this is for the
analyst to reflect on the conclusions
6. Sensitivity Analysis
--- Page 83 ---
Copyright Christopher Crowley
Security Operations
83
Analysis of Competing Hypotheses Background
Malware drive by
download attempt
Phone
actually
infected
Targeted not from
site, but from
network user is on
Site Accessed when
ad showed
5 2 3
Bad Grammar 3 3 3
Compulsion to risky
action
3.5 3 3.5
Advertised only on
one site
7 1 2
Threatening? 3.5 3 3.5
Reputation weight of
domain?
4 3 3
Total: 26 15 18
--- Page 84 ---
Copyright Christopher Crowley
Security Operations
84
Analysis of Competing Hypotheses ACH Applied
- Write a brief explanation of your results (potential for
automation and orchestration here)
- Identify most likely hypothesis
- Identify your assumptions
- Identify the sensitivity of your analysis (what does it depend
upon being true?)
- Identify other hypotheses considered but dismissed
7. Report Conclusions
--- Page 85 ---
Copyright Christopher Crowley
Security Operations
85
Analysis of Competing Hypotheses ACH Applied
- Identify a few things that are likely to happen in the
future, based on what you’ve hypothesized is the
most likely explanation
- This isn’t easy. For example, if you hypothesized an
intruder has compromised a host, there will likely be
subsequent reconnaissance and pivoting attempts
using exploits and password guessing
8. Milestones
--- Page 86 ---
Copyright Christopher Crowley
Security Operations
86
Analysis of Competing Hypotheses ACH Applied
- Give your completed work to your partner analyst
- This isn’t escalated to a higher tier, done by a peer analyst
- The partner should be able to understand what you wrote
and your explanation without you explaining it to him/her
– repeat: you’re not allowed to explain the report
to your peer. Peer reads it, criticizes it, you improve it
- Don’t agree, only point out problems that are present, this
is what makes analysis better: criticize to strengthen
9. Peer Review (Crowley added)
--- Page 87 ---
Copyright Christopher Crowley
Security Operations
87
Analysis of Competing Hypotheses Background
- 1) Identify hypotheses – brainstorm
- 2) List attributes – for and against each hypothesis
- 3) Build matrix – with hypotheses at top, evidence on side
- 4) Refine matrix – remove or consolidate as appropriate
- 5) Tentative conclusions – disprove (don’t validate) item
- 6) Sensitivity – change critical elements to assess
- 7) Report conclusions – relative likelihood of each
- 8) Milestones – future events suggesting error in analysis
- 9) Peer Review
Review - ACH – 8-Step ACH Process
--- Page 88 ---
Copyright Christopher Crowley
Security Operations
88
Analysis of Competing Hypotheses Background
- We can create templates for the common situations
we encounter to expedite the analysis portion
- There’s a danger to this: you are not supposed to just
fill in the template – don’t be limited by it, it’s a start
- Work on templates, and revise perpetually
- Start now: 3 most frequent situations requiring
analysis, and build templates for those over the next
three weeks
Templates – Benefits and Risk
--- Page 89 ---
Copyright Christopher Crowley
Security Operations
89
Analysis of Competing Hypothesis
Incident Type Hypothesis 1 Hypothesis 2 Hypothesis 3
…
Data 1
Data 2
Data 3
Data 4
…
Template
Richards J Heuer – Analysis of Competing Hypothesis
Book Download Available
Probably a good idea to develop something like
these templates for use in your environment as
an analytical tool. It forces externalization and
objective practice. Important – templates a
work expediter, not a checkbox to fill in.
--- Page 90 ---
Copyright Christopher Crowley
Security Operations
90
Conclusion
--- Page 91 ---
Copyright Christopher Crowley
Security Operations
91
Thank You
- CCrowMontance (twitter)
- https://www.mgt517.com/soc for this slide deck & other
public decks, plus additional references
- Redistribution authorized, but please provide citation
- https://www.montance.com/soc/timeline : current project
for building a SOC
- MGT517? :https://soc-class.com &
https://montance.blogspot.com/2019/04/security-
operations-class-status.html
I Appreciate Your Time and Attention
Related Q&A
- Q: What was the root cause of the Equifax breach?A: Failure to patch a known vulnerability in Apache Struts.
- Q: What specific vulnerability was exploited?A: CVE-2017-5638.
- Q: What was the 'detection deficit' in the Equifax case?A: Approximately 76 days.