Hacking Reality: Why Data Science Is the Blue Team’s Ultimate Exploit

Author’s Note

This article is part of my research exploring quantitative methods for information security risk management. The views presented in this blog are entirely my own and do not represent the views of any organization or other affiliation.

Interpretability or Predictability?

Cybersecurity is full of models: threat models, maturity models, risk models, severity models, control models. We are not short on frameworks.

What we are short on is something more awkward and scientific: closing the loop. Measuring whether our models predict the outcomes we care about, and whether decisions informed by those models measurably reduce risk. This may seem intuitive, but in order to evolve into a scientific discipline, we must explicitly validate our outputs.

The Two Cultures of Statisticians

In 2001, a statistician named Leo Breiman published a divisive paper that split statisticians into two cultures of modeling complex system behavior.

Culture One: Data Modeling

The data model culture assumes reality works according to an interpretable mechanism. You observe events, hypothesize a structure that minimizes the distance to a known data shape/relationship, and then use that fitted model to explain and build forecasts.

Culture Two: Algorithmic Modeling

The algorithmic model culture treats the mechanism as unknown. You don’t pretend to know how reality generates its outputs. Instead, you search for a function that predicts those outputs accurately, and you let the data reveal what matters.

Breiman’s argument was straightforward: nature is complex, and models do not need to follow known patterns. He later invented the random forest algorithm to produce models of complex systems, which is a core underpinning to modern machine learning.

Both cultures had one thing in common that modern information security models lack: measuring how well the model performs against our desired outcome.

But both cultures share something that much of cybersecurity modeling still lacks: explicit measurement of model performance against outcomes.

In other words: validation.

Pre-Validation Modeling: The Third Culture

Cybersecurity does validate many things locally: detection coverage, red team results, penetration tests, etc. But as an industry, we still rarely validate end-to-end risk models and risk-informed decisions against measured outcomes in a standardized, repeatable way.

We’re essentially running an open-loop model of cybersecurity. Because we operate in a low validity environment, we produce risk scores, labels (looking at you classic stop light/risk matrix model of labeling findings Critical, High, Medium, Low), and narratives that make us feel good about what we observe. We then translate those observations into actions, but we do not take the last crucial step: measuring whether those actions actually reduced the outcomes we claim to be managing.

We are entrenched in what we can deem the third culture, a state of our discipline that does not scientifically validate if our defensive state is reducing organizational risk.

I argue that we need to move the industry out of this way of thinking and towards validation-based models.

Using vulnerability management’s progression into data science as a key example, I hope you read this post and come away with an understanding of why this shift matters and thinking about what the bridge between the three cultures can look like in practice.

Vulnerability Management as a microcosm

I’m biased because I am currently a practitioner in this space, but Vulnerability Management’s lack of validation is what drew me to the space in the first place.

Consider how the industry currently thinks about vulnerability risk. I’m not going to belabor the flaws of our current approaches, as they have been covered extensively in works such as “How to Measure Anything in Cybersecurity Risk” (The Green Book), “Measuring and Managing Information Risk: A FAIR Approach” (The FAIR Book), and other industry research and commentary. And I’m not calling these out as being “wrong.” These models are extremely important stepping stones showing that our work is not “done.”

But to summarize:

CVSS
The Common Vulnerability Scoring System (CVSS), is a widely used severity model and treated as an allegory for risk by some. It assumes that vulnerability characteristics (attack vector, complexity, CIA impact, etc.) contain sufficient information to classify severity, and severity then becomes a proxy for “what we should do first.”

CVSS can be useful, but it’s not built as an outcome-validated model. It is not designed to learn whether its ranking correlates with exploitation, loss, or harm for your environment. The environmental labels that can be used as an input to influence the score of a vulnerability are a non-starter because of the arithmetic done on the ordinal rankings of classifications, which is a big no-no.

SSVC
Stakeholder-Specific Vulnerability Categorization (SSVC) is another risk model in the form of a decision tree. Five parameters, 72 possible input combinations, four discrete outputs: Track, Track*, Attend, Act. It’s cleaner than CVSS and more defensible. And it applies the exact same logic to the millionth vulnerability as it does to the first. It doesn’t learn. It doesn’t update. It doesn’t converge on anything. It also suffers from what Jay Jacobs has termed “the last mile” problem, where decision tree output is transformed into an action to address vulnerabilities “because it intuitively feels right.” It does not validate if the action taken actually reduced organizational risk.

FAIR
The Factor Analysis of Information Risk (FAIR) ontology is the most sophisticated and defensible of the current risk models in cybersecurity. It decomposes risk into Loss Event Frequency (LEF) and Loss Magnitude (LM), structures expert elicitation into probability distributions, and produces Monte Carlo simulations of financial loss. Even though I’d classify it as pre-validation, it is statistically coherent and uses elements of data models (assumed distributions based on generalized trends), but it lacks a standardized approach to validate its outputs locally for practitioners making use of it. As Jack Jones rightly points out, empirical validation is a known limitation of FAIR and one we should continue to push forward on. The time scales FAIR-based risk models deal with can make them prohibitive to empirically validate, but that doesn’t mean we can’t validate the internal model structure.

Towards Algorithmic Models: Hacking Reality

EPSS

The first version (v1) of the Exploit Prediction Scoring System (EPSS) was a data model that used logistic regression to model the relationship between 16 vulnerability attributes and the vulnerability’s likelihood to be exploited in the wild. It used validation mechanisms such as the area under the curve (AUC) of the Precision/Recall curve to show how well it was predicting the future, and while not perfect, it showed the industry that it’s possible to use statistical modeling to refine and advance the industry’s approach to vulnerability management, which helps address the rising arrival rate problem of known vulnerabilities for the defender.

EPSS has since evolved into a culture two model, utilizing an algorithmic mechanism (XGBoost*) to produce an algorithmic model that hacks each attribute’s contribution to future exploitation. It’s a better guess of exploitation reality than the previous versions, and improves on the flaws holding the first version back. EPSS didn’t just improve vulnerability scoring. It demonstrated that cybersecurity could move from assumed models to learned ones.

The creators and model itself don’t assume to know why vulnerabilities get exploited, but rather they let the data explain their relationship to exploitation. Instead of offering a competing theory for vulnerability management, its creators (Jacobs, et al.) decided to utilize algorithmic modeling tools to discover the hidden relationships between vulnerability attributes and their predictive power. It embraced Breiman’s second culture: treat the mechanism as unknown, optimize for prediction.**

*XGBoost builds decision trees sequentially (boosting), where each new tree corrects the errors of the previous ones, typically offering higher accuracy and performance, particularly on complex, large-scale data. It is a cousin of Breiman’s random forest algorithm.

**Detractors will say the reality that EPSS is modeled against isn’t the full picture. Nobody has a full picture, and this is precisely why we need to model. Because it is built on feedback, EPSS will only get better as more data sources and features are added. The hope is that it will inspire more public algorithmic models across cybersecurity.

The Ultimate Exploit

The word “hacking,” stripped of its cultural baggage, describes a specific cognitive operation: building a model of a complex system that’s accurate enough to predict and validate exploitable states. A hacker doesn’t need to understand every line of code in your application. They need a model of the system’s behavior that’s good enough to find an exploitable state the designers didn’t anticipate: a state where the system does something it wasn’t supposed to do.

This is exactly what algorithmic modelers do. They don’t need the true data-generating mechanism. They need a function that approximates reality well enough to make accurate predictions about exploitable states. The system is a black box. The ability to model chained exploitable states are what give hackers their power.

This leads us to our ultimate exploit. Blue teams have home field advantage, which means we have the information at our fingertips to produce threat models, risk models, and other models of reality that drive decisions. But, validation mechanisms remain a key component that is missing in most industry standard best practices. We can exploit our advantage by building localized algorithmic models that perform better than current frameworks using statistical validation mechanisms. All we need to do is start intentionally collecting the outcome data.

Why This Matters Now

From my vantage point, two forces are driving model maturation urgency. Again, sticking with vulnerability management as a key microcosm.

The Flood

Vulnerability arrival rates are accelerating. CVE publications grew 164% from 2020 to 2025, hitting roughly 48,000 disclosed vulnerabilities. FIRST’s 2026 forecast projects a median of approximately 59,000 CVEs this year, with extreme-but-plausible scenarios reaching 118,000. The volume problem isn’t that there are more dangerous vulnerabilities. It’s that disclosed vulnerabilities are scaling faster than human remediation capacity.

The vulnerability arrival rate problem has a root cause. Jen Easterly identified it during her tenure as CISA Director and articulated it most forcefully in Foreign Affairs in late 2025: the United States doesn’t have a cybersecurity problem, it has a software quality problem. Easterly’s Secure by Design initiative and Bob Lord’s work as its chief architect represent a generational bet on fixing the source, reducing the arrival rate of exploitable defects through vendor accountability and software liability reform.

Algorithmic models such as EPSS are crucial for information security folks to remain above water in a sea of signal-to-noise complexity while the stellar folks I mentioned above work to address the upstream economics.

The Adversary

Human (and now AI agent) adversaries are natural algorithmic modelers. They don’t theorize about your defenses. They probe, observe, adapt, and optimize. They create intuitive or explicitly-defined models of target system behavior to decide attack paths they predict will lead to their goal or highest expected value. They don’t need interpretability; they need working prediction.

Autonomous exploitation agentic systems act in a similar way: they model the attack surface and make determinations about which paths will be the most likely to yield an exploitable state of the system. Recent news and research about agentic exploitation make this transition towards validation and algorithmic modeling all the more urgent.

If defenders remain attached to interpretability as the primary virtue while attackers optimize for predictability, defenders will lose by choosing the wrong objective function.

Wrapping up

The ultimate hack isn’t a zero-day. It isn’t a novel exploit chain or a clever phishing lure. It’s engineering a model of reality that’s accurate enough to make rational decisions under uncertainty. That’s what data science gives defenders. That’s what algorithmic modeling makes possible. And that’s the exploit that blue teams need to arm themselves with.

The organizations that engineer the observation infrastructure now (the structured, labeled, time-series data that that supports validation loops for algorithmic models to eventually consume) will have a compounding advantage over those remaining with unvalidated, open-loop risk models. The data is already flowing through firewalls, SIEMs, endpoint agents, identity logs, and vulnerability scanners. It’s partially structured, labeled-ish, and sitting in silos. But it’s there. The question is whether we’ll build additional observation architecture with intention to capture outcome labels and trace decisions that enable real validation.

Because if we don’t build a better model of ourselves, our adversaries will.
And they won’t care whether it’s interpretable.

Stephen Shaffer is a Principal Security Engineer at Moderna, Co-Chair of the FIRST EPSS Special Interest Group (SIG), and a member of the Enterprise Risk Quantification Institute (ERQI).

Interested in joining the EPSS SIG? Request to join here.

Bonus Thoughts

If all of this sounds abstract, look at where AI research itself is heading. We’ve already seen AI agents have success with discovering new vulnerabilities and their associated exploit code. And outside of cybersecurity, Yann LeCun’s work on Joint Embedding Predictive Architectures (JEPA) is building toward what he calls world models: AI systems that learn internal representations of how complex environments behave, predicting future states not by reconstructing every pixel or token but by operating in abstract representation space, capturing the features that matter and discarding the noise that doesn’t. His late-2025 LeJEPA paper shows the trajectory: systems that build increasingly rich internal models of reality through prediction alone.

This is Breiman’s second culture taken to its logical, radical endpoint. Don’t model the mechanism. Don’t even model the raw outputs. Learn to predict abstract representations of future states, and let the structure emerge from the prediction task itself. It’s light-years ahead of where cybersecurity risk modeling sits today, but I wouldn’t be surprised to see this applied to cybersecurity “real soon.” ***

*** I have no idea if/when this will happen.

That being said, the future of addressing cybersecurity risk comes down to a simple question:

Do we care more about the interpretability of a model, or its predictability?

What else is cooking?

Stay tuned for a continuation of this blog, either in the form of another post or a whitepaper. Synthesizing your own environment’s vulnerability risk using validation is within reach.