This is the long-form version of a short panel talk I gave this week, on cyber resilience and assurance in an AI-driven world. My background here is as a practitioner — I work with EmilyAI in the defence space — and the argument I wanted to make on the panel is that a lot of our instincts about resilience and assurance, good instincts honed over years, do not quite transfer once AI is in the mix.
I put it as five shifts. Not five problems to be solved by Friday — five changes in how we have to think. They apply well beyond defence, but defence is where the edges are sharpest, so that is where I will draw the examples.
One — assure the model, not just the infrastructure around it
We are good, in this industry, at asking "is this system secure?" We are much less practised at asking "can I actually trust what the model just told me?"
That gap matters because our whole accreditation machinery assumes deterministic behaviour. Give a system the same input, expect the same output — that is the premise a safety case, an assurance case, an accreditation sign-off is built on. Models do not behave like that. Same input, different day, different sampling, and you can get a different answer. The property we have always leaned on is simply not there.
So the idea of an assurance case has to evolve rather than be discarded. What does it mean to assure something probabilistic? In practice it means three things we have not traditionally demanded. Provenance of the model — where it came from, what it was trained on, who touched it. Explainability of the output — not a philosophical account of cognition, but enough of a reason that a human can sanity-check it. And understood bounds — a documented, tested envelope of where the model is reliable and where it falls apart.
In defence we have never fielded a capability we could not assure. That is not a slogan; it is the deal. AI does not let us out of that deal. It forces us to redefine what assurance means, and to build the evaluation harnesses, the red-teaming, and the known-failure catalogues that a probabilistic assurance case actually needs. If you cannot describe the conditions under which your model is wrong, you have not assured it. You have just been lucky so far.
Two — AI is both the threat and the shield
The second shift is to be honest that AI sits on both sides of the wire at once.
On the threat side, attackers are automating the boring, labour-intensive parts of their job. Reconnaissance at scale. Phishing that is fluent in your language and your org chart. Malware that rewrites itself to slip past signature detection. Deepfakes aimed squarely at our own people — the finance clerk, the duty officer, the person on the end of the phone at 3am. None of this is new in kind. All of it is new in volume and cost.
On our side, the same technology gives us detection and response at machine speed — catching the anomaly no human watch-floor could ever spot in the noise. That is real, and it is worth having.
But there is an uncomfortable asymmetry underneath it, and it is the oldest asymmetry in this field made worse. The defender has to be right every single time. The attacker just needs the model to find the one gap. AI widens that gap because it lets the attacker try far more often, far more cheaply. So resilience becomes, increasingly, a question of who operationalises this faster — and, the part people skip, more responsibly. An automated response system that acts at machine speed will also make mistakes at machine speed. Speed without judgement is just a faster way to be wrong.
Three — the attack surface now runs upstream, into the model itself
The third shift is the one I think gets missed most often. The attack surface has moved upstream, into the model itself.
Data poisoning. Compromised open-source models pulled from a public hub. Tampered weights. Back-doored fine-tunes. The supply chain now includes the training pipeline, and the training pipeline is a place most organisations have no visibility into at all. We spent the best part of a decade learning how to secure the software supply chain — dependency pinning, signing, provenance, the whole apparatus. We are, more or less, starting from scratch on the machine-learning supply chain.
The hard question for anyone building on top of AI is simple to ask and awkward to answer: do you actually know the provenance of the foundation model your capability now depends on? Who trained it, on what, with what integrity controls, and can you prove the weights you are running are the weights they published? For most teams the honest answer today is no. Model cards are not signatures. A download count is not assurance.
Cyber resilience used to stop at the network edge. Then, over the last few years, we accepted it had to reach into the software supply chain — the compromised dependency, the poisoned build. AI extends that same logic one step further back again. Resilience now has to reach all the way to where the model was born.
Four — the accountability gap in human-machine teaming
Fourth is human-machine teaming, and the accountability gap sitting inside it.
"Meaningful human control" is a phrase that sounds settled until you put a real person next to a confident machine. Automation bias is not a fringe effect; it is how people behave. We trust the confident output. We trust it even when it is wrong, and we trust it more when we are tired, busy, or under pressure — which in our world is most of the time.
And when an AI-assisted decision does go wrong, who owns it? The operator who acted on it? The developer who trained the model? The accrediting authority who signed it off? We do not have clean answers yet, and pretending we do is how you end up with a human "in the loop" whose only real function is to absorb the blame for a system they were never equipped to overrule.
For me, resilience here means graceful degradation. The system has to be able to recognise when it is out of its depth and hand control back to a human cleanly — with enough context that the human can actually take over, not just inherit an alarm. That means calibrated confidence, not false precision. It means designing the handback as carefully as we design the capability. Because in our domain these decisions carry consequences you cannot undo, the human has to be genuinely in the loop — not nominally present to countersign a machine they have been trained to defer to.
Five — sovereignty, and running AI at the edge
The fifth shift is about where the AI actually runs.
The most capable models live in commercial clouds, behind an API, in someone else's data centre, often in another jurisdiction. The operational reality for a lot of us is the opposite of that: disconnected, contested, and classified environments. The two do not meet in the middle on their own.
So the challenge is concrete. How do you run genuinely useful AI without leaking sensitive data offshore, and without assuming you will even have connectivity when it matters? That pushes hard towards sovereign AI, air-gapped deployment, and serious attention to data residency — not as compliance box-ticking, but as a design constraint from the first line. It also pushes towards smaller, quantised models that can run on the tin you can actually deploy, and towards being honest about the capability you lose when you leave the big cloud model behind. That trade is real, and it is better made deliberately than discovered in the field.
The battlefield does not come with five-nines of uptime. Neither does a disaster zone, a ship at sea, or a site mid-incident with its links cut. Our AI has to be resilient to the environment, not just to the adversary. A model that only works when the network is up is not a resilient capability. It is a demo with good manners.
The thread through all five
So — five shifts. Assure the model, not just the infrastructure around it. Treat AI as both threat and shield, and mind the asymmetry. Follow the attack surface upstream into the model and its supply chain. Close the accountability gap in how humans and machines team. And take sovereignty and the edge seriously.
The thread running through all of them is the same, and it is the thing I actually wanted people to leave the room with. Resilience and assurance are not things we bolt on at the end any more. They never really were, but AI removes the option of pretending otherwise. With these systems, resilience and assurance have to be designed in from the very first decision — the choice of model, the shape of the pipeline, the terms of the human handback, the place it will run.
Bolt-on security has always been the expensive kind. With AI it is not just expensive. It does not work at all. That is the shift underneath the other five, and it is the one worth arguing about.