_Post 6 of the AI in cyber series._

The CrowdStrike Falcon outage in July was not an AI incident. It was a content-update incident — a faulty channel file that caused a kernel-mode crash on 8.5 million Windows machines. The blast radius was extraordinary because CrowdStrike Falcon is, on the machines it protects, running with kernel privilege and updating itself automatically from CrowdStrike's central distribution. The same shape of risk applies, with sharper edges, to the AI-in-security category as it is currently being built.

This post is about why single-vendor concentration of intelligent agents is structurally riskier than single-vendor concentration of conventional security tooling, and what architectural defences are worth building before the AI version of this incident arrives.

The blast radius the July outage made visible

A specific scenario that is worth keeping in mind through the rest of this post.

On the morning of 19 July 2024, a content update from CrowdStrike's central infrastructure took simultaneous effect on every Falcon-protected machine in the customer base. The update was bad. The result was a global outage that affected airports, hospitals, banks, broadcasters, supermarkets, and emergency services. The recovery, in many cases, required hands-on local intervention on each machine.

The structural lesson — covered in the retrospective post — is that having most of your endpoint security on a single vendor's automatically-updated kernel-mode agent gives that vendor effective remote-command authority over your estate. The vendor is not malicious. The vendor had a very bad day. The effect was the same as if a malicious actor had compromised the update channel.

Now apply the same model to AI in security. The vendor's AI agent is, in the current product framings, running on a similar update cadence — the underlying model is updated centrally, the agent's behaviour can change between this morning and this afternoon, the customer cannot meaningfully audit what the model is doing differently today. The blast radius of an AI update going wrong is at least the same as a content-update going wrong.

Why the AI version is structurally worse

Three reasons.

The AI is making decisions, not just detecting indicators. A conventional security agent matches known patterns and signals to a central pipeline. An AI agent makes verdicts. When the central model is updated, the verdicts produced today are not the verdicts that would have been produced yesterday. A wrong content update breaks detection. A wrong model update changes interpretation. The wrong-interpretation case is harder to detect than the wrong-detection case because there is no clean baseline to compare against.

The AI's reasoning is opaque. A signature-based detector that is producing false positives is debuggable in minutes. An LLM-based detector that is producing different verdicts after an update is debuggable in hours, sometimes days, and the answer is often we cannot fully explain what changed. The blast radius is the same; the mean-time-to-understanding is longer.

The customer has no path to a known-good version. The CrowdStrike content-update model at least supports a deployment-ring approach in principle — early canaries, staggered rollout, ability to pin a known-good version. The hosted-LLM model does not yet have a comparable customer-facing control. The customer cannot say give me yesterday's model in most cases. The vendor's roll-forward becomes the customer's only path.

What EmilyAI's architecture does about this

The architectural decisions we made in 2018 were not anticipating the LLM era. They were anticipating a different version of the same problem — what if the model is wrong, and how do we handle the rollout safely. The decisions turned out to be appropriate for the LLM-era concerns too.

Champion-and-challenger on the inference tier. EmilyAI's two L40S accelerators carry the production champion model on one card and a candidate challenger on the other. The challenger sees the same live traffic as the champion, produces verdicts on a shadow basis (not delivered to the customer), and lets us measure the divergence between champion and candidate before promoting the candidate. A bad model never reaches production traffic.

Replay safety. Hours of recent traffic remain in Redis Streams and can be re-processed through any historical or candidate model without re-fetching from upstream SIEMs. A regression that is detected after deployment can be diagnosed against the same input stream that produced the regression, against the previous model version. We can compare what would yesterday's model have said about this morning's traffic.

The model registry pins specific binaries. Every verdict references a specific compiled model artefact by cryptographic hash. Roll back to yesterday's model is a single-command operation that produces, by construction, bit-identical results to what was running yesterday.

Single-tin deployment isolates blast radius. The on-premises deployment topology means that even a catastrophic vendor-side incident at UKCD does not reach the customer's deployment. Each customer's EmilyAI runs on their own R760; failures do not propagate.

These are not novel ideas. They are the standard discipline for any system that is making consequential decisions and that needs to support rollback. The LLM-era products have, in some cases, been built faster than the equivalent discipline could be retrofitted.

What customers should ask their AI security vendors now

A short list of questions that are worth pushing on, particularly in any procurement decision through late 2024 and into 2025.

Can I pin the model version that handles my traffic, and roll back to a previous version if necessary? If the answer is no, you do not have the equivalent of yesterday's working configuration.

What is the deployment-ring discipline for model updates? Canaries, staggered rollout, rollback procedure. If the vendor cannot answer this, the model gets pushed to your tenancy on the same cadence as the rest of their customer base, with the same blast-radius implications as the CrowdStrike content-update model.

What does the audit trail look like for a verdict? Which model artefact, identified specifically. We logged that the model ran is not the same as we logged that model artefact X produced output Y on input Z.

What is my disaster scenario? A vendor-side model regression that affects every customer at once, including yours. How long until you know? How do you respond? What is your independent ability to continue operating?

These questions sound paranoid until you remember that they would all have been useful to ask CrowdStrike, a respected vendor, before 19 July 2024.

The wider question

The CrowdStrike outage produced a small but real industry-wide conversation about single-vendor concentration in endpoint security. The conclusion was, broadly, that having all your security on a single vendor's auto-updated kernel-mode agent is a resilience risk worth modelling, even though most customers will continue to do it because the operational economics favour it.

The equivalent conversation about AI in security has not yet happened in the public industry voice. It will. The shape it will take is, I think, a debate about whether the AI agent should be operated by the vendor (with all the central-update implications) or by the customer (with all the operational-burden implications). Both shapes are defensible. The current market default — operated by the vendor — is the higher-blast-radius shape and is the one the CrowdStrike incident should have prompted more public conversation about.

What is next

In six weeks: the reasoning models — OpenAI's o1 was announced in September, and the thinking model category is what most of the AI conversation will be about through the rest of 2024. What it changes for SOC work, and what it does not.