Cisco has published two significant IOS security advisories this month. Both affect widely-deployed router platforms; both are exploitable remotely; both have produced operational incidents at organisations slow to patch.
IOS — Cisco's router operating system — is not Linux's IOS. It is the proprietary OS that runs the routers on which the internet depends. The security properties of IOS are essentially closed; operators do not have access to the source. This produces a different relationship with vulnerabilities than the open-source operations I usually write about.
The two advisories, briefly
CSCdv47948 — TCP small-buffer denial of service. A specifically-crafted sequence of TCP packets sent to a Cisco router causes the router's buffer pool to be exhausted. The router becomes unresponsive to most traffic until manually restarted. Affects multiple IOS versions. Patches available.
CSCdv11106 — HTTP server vulnerability. Many Cisco routers run a small HTTP server for web-based configuration. A specific URL pattern causes the HTTP server to crash, possibly with arbitrary code execution. Affects routers with the HTTP service enabled (which is many; it is on by default on most platforms). Patches available.
Both are serious. Both affect production internet routers globally. Both produce operational incidents in environments where patching the router infrastructure is operationally difficult.
Why patching routers is hard
A few specific reasons that make router patching different from server patching.
Restart required. Most IOS patches require a router restart to take effect. The restart causes a brief network interruption — typically 30-90 seconds. For a router carrying production traffic, this is a real cost; the patching has to be coordinated with downstream operators, scheduled in maintenance windows, sometimes negotiated with customer SLAs.
Limited testing capacity. Most organisations have one or a handful of routers in any given role. Testing a patch in a non-production environment is often impossible because the production environment is the only environment. Patches sometimes turn out to introduce regressions that are only discovered after deployment.
Long support tails. Cisco supports IOS versions for many years. An organisation running an older version is supported but may not be on the same patch schedule as current versions. Vulnerabilities that affect multiple IOS versions produce a complex coordination problem about which patches to apply where.
Configuration coupling. Some patches require small configuration changes to behave correctly post-patch. The configuration changes may affect things other than the patched feature, and the cascade can be hard to predict.
Vendor support relationships. Many organisations cannot patch routers without explicit approval from their Cisco support contract. This adds days or weeks to the patching cycle.
The net effect: router patching is, in most environments, slower than server patching by an order of magnitude or more. The vulnerability window is correspondingly longer.
The closed-source aspect
This is where the IOS situation diverges most sharply from open-source infrastructure.
The vulnerability disclosure is mediated by Cisco. Operators learn about IOS vulnerabilities when Cisco publishes an advisory. The technical details Cisco chooses to publish are limited; the underlying root cause is often not described in detail. Operators have to take Cisco's word that the patch addresses the issue completely.
Independent verification is difficult. Without source access, security researchers cannot independently verify that a patch is correct, that it does not introduce new bugs, or that it covers all variants of the vulnerability. The trust model is centralised at Cisco.
Workarounds are limited. For an open-source product, an operator who cannot patch immediately can sometimes apply a custom mitigation — modifying source, configuring around the bug, deploying a wrapper. For closed-source IOS, the operator has whatever workarounds Cisco documents, and nothing else.
Long-term posture is opaque. The structural improvements that an open-source project's maintainers would make are not visible. Cisco may be aggressively cleaning up codebases; they may be in a steady-state pattern of patching specific bugs. Operators have no way to know.
This is not a complaint about Cisco specifically. It is a structural observation about closed-source infrastructure generally. The same observations apply to most commercial firewall products, most enterprise software, most embedded-systems vendors.
What this teaches about reliance on closed infrastructure
A few observations.
The trust model is binary. Either you trust the vendor to handle security competently, or you do not. There is no middle ground where you partially trust them and verify the rest independently. Open-source provides this middle ground; closed-source typically does not.
Switching costs amplify the trust. A network that is built on Cisco IOS is, structurally, committed to Cisco's security practices for the lifecycle of the equipment (typically 5-10 years). Switching to an alternative — Juniper, Foundry — is operationally substantial. The commitment is long.
Diversity helps. A network that has both Cisco and Juniper or Cisco and a Linux-based router is, structurally, more resistant to the specific class of vulnerability that affects only one vendor. The argument for platform diversity applies to network infrastructure as much as to host operating systems.
Open alternatives are emerging slowly. Linux-based router distributions are operationally credible for small to medium deployments. They are not yet competitive with commercial routers for very high traffic levels but are closing the gap. By 2003-2005 I would expect them to be a real option for many deployments.
What I am doing
For my own home setup: my perimeter is OpenBSD, not Cisco. I am not directly affected by these advisories. The OpenBSD project's structural advantages over closed-source alternatives are exactly the kind of thing I have been writing about all year.
For friends with Cisco infrastructure: I have been sending advisory notifications and helping with patch scheduling. The friend at the small consultancy whose October incident I wrote about has multiple Cisco devices; we coordinated the patches and the maintenance window.
For my Snort sensor: I have added rules for the specific HTTP-server exploitation patterns. This catches scanner activity from sources looking for vulnerable Cisco routers; the same patterns are useful as proxy for general Cisco-targeting attacker activity in my range.
The structural observation
Closed-source critical infrastructure is a strategic vulnerability for the operator. The trust model is irreducible; the visibility is limited; the alternatives are constrained by switching costs.
This is not a new observation; it has been made by many people for many years. The IOS advisories this month are a quiet reminder that the observation continues to hold. Operators who can deploy open-source alternatives have a structural advantage that compounds over time. Operators who cannot are accepting the trust trade-off; the trade-off is real and is sometimes the right answer, but should be made deliberately rather than by default.
For my own writing: more of my year-end and 2001 posts are likely to address this. The closed-versus-open question is one I have not written about explicitly enough; the IOS situation is a useful prompt to do so more directly.
More as the year wraps up.