I have spent a substantial portion of this year writing about patches: the BIND advisories, the wu-ftpd backlog, the Microsoft IIS rolling story, the relentless cadence of Bugtraq posts about new bugs in old software. Each patch closes a hole. The discipline of "keep patched" is the single most useful operational habit in this entire field.
It is also, increasingly clearly, not enough on its own. I want to write about why.
The premise of patching
The model behind patching is simple. Vulnerabilities are bugs. Bugs get reported. Vendors ship fixes. Operators apply the fixes. The hole is closed. Move on to the next.
This is, on the strict theory, a complete defence. If every operator patched every vulnerability instantly, the only successful attacks would be against not-yet-discovered bugs (zero days) or against operator error (misconfiguration).
Almost none of this is true in practice.
Where the model breaks
Four ways, in order of how often they bite.
There is always a window. Between vulnerability disclosure and patch deployment, every host running the affected version is exploitable. The window is measured in days for the most diligent operators, weeks for most, months for the long tail. During that window, mass-scanning attackers will find and exploit hosts that are otherwise well-managed. The patching-discipline operator is not magically protected during the window — they are exposed like everyone else, just for a shorter time.
Patches are themselves a code change. Every patch can introduce new bugs. Most do not. Some do. The June BIND patch that I was so eager to apply turned out to interact badly with one of my zone configurations and produced an obscure logging bug that took three days to track down. The patch was correct; my deployment was wrong; the consequence was real. Patching at speed produces these incidents in proportion to the patching speed.
Patched code can still be exploitable in your environment. This is the one I find most unsettling. A patch addresses the known bug. Other bugs — same code path, similar pattern, different trigger — may exist in the patched code. "Patched against the current advisory" does not mean "unable to be compromised through this code path".
The IIS patch sequence is an instructive example. Each advisory has been against a specific bug. Within a few months of each patch, a related bug is found in the same component. The component itself is buggy by structure; individual patches do not change that.
Patches do not fix architectural problems. Many vulnerabilities are not bugs in the strict sense — they are properties of the design. A service that accepts unauthenticated connections from anywhere on the internet does so by design; the patch does not change that. The protection against DDoS is not a patch. The protection against credential brute-forcing is not a patch. The protection against the worker-bee user opening a Word macro is not a patch — it is a structural property of how documents and applications interact.
What this implies
A few things, written for my own future use.
Patching is necessary but not sufficient. It is a baseline discipline. The defenders who only patch are doing the minimum and pretending it is the maximum. Patching is the floor of the ladder, not its rungs.
The other rungs are mitigation, isolation, and monitoring. A bug that has not been patched can still be unreachable through firewall configuration. A bug that is exploited in a service can still be contained by chroot and unprivileged user IDs. A successful exploit can still be detected and contained by Snort and log monitoring before it spreads further.
Defence in depth is not an empty phrase. It is the recognition that any single layer can fail and the architecture should still hold.
The asymmetry is psychological. It is much easier to describe a security programme as "we patch promptly" than as "we patch promptly and run structured logs and maintain egress filters and run quarterly pen tests and have a working incident-response procedure". The first sentence fits in a single PowerPoint bullet. The second is a programme.
The shorthand of "we patch" is the ideological enemy of doing the rest. It produces the comfortable feeling of having done the necessary thing, when in fact you have done only the easiest part.
A small case study
Last week I had a small incident on my own infrastructure. A friend with shell access on one of my machines had reused the password from a different service that had been compromised. The attacker tried the credential, got in, looked around, saw the kind of constraints I have built up — limited capabilities, no useful root path, no interesting data accessible — and went away.
The incident was visible because the structured logs recorded the unusual login pattern; the log scanner caught it within a few hours; I was able to confirm by reading the logs that the attacker had not gone deeper than the initial shell. I disabled the friend's account, rotated his SSH key, and told him to change his password elsewhere too.
This incident had nothing to do with patching. The system was fully patched. The vulnerability was a credential reuse attack that no patch can address. The defence — early detection, structural containment — was the entire response. If my discipline had been only "keep patched", the attacker would have had unrestricted shell access and possibly more.
The cumulative position
A year of writing about patches has, paradoxically, made me think they are less central than I had assumed. They are still the most important single thing. They are not the only important thing, and pretending they are produces the kind of programme that fails on the first non-patch problem.
The operators I respect most are the ones who patch reflexively but talk about it least. The ones who only talk about patching, in my experience, do less of all the other necessary things.
For the year ahead, my own personal target is to spend roughly a quarter of my security time on patching, a quarter on monitoring and detection, a quarter on architectural improvements, and a quarter on testing and learning. The exact ratio is arbitrary. The principle that patching gets a quarter rather than a half is the part that has changed.