A week after Heartbleed's public disclosure and the engagement work has been almost continuous. The vulnerability — CVE-2014-0160, a buffer-overread in OpenSSL's TLS heartbeat extension that allows arbitrary remote read of up to 64KB of server memory per query — has been in the wild for approximately two years (introduced in OpenSSL 1.0.1, December 2011) and was simultaneously discovered last month by Neel Mehta at Google and by Riku Hietamäki, Antti Karjalainen, and Matti Kamunen at Codenomicon. The patch shipped on the seventh; the Codenomicon team published the heartbleed.com explanatory site simultaneously, branding the vulnerability and producing the cleanest piece of public-facing security communication I can remember for an issue of this scale. The structural significance is the part I want to write down before the operational firefighting overwrites it.
Heartbleed is the moment that the structural argument I have been making about commercial encryption infrastructure stops being academic. The vulnerability sits in OpenSSL — the open-source TLS implementation that underlies a substantial proportion of public-facing internet services, somewhere between sixty and seventy per cent of the secure-HTTP web servers on present scans, plus uncountable other TLS deployments — and is exploitable remotely without authentication, leaks server memory including private keys, and has been doing so since 2011. The implication is that any TLS-protected service running affected OpenSSL versions has, for the past two years, had the possibility that its private keys are now in adversary hands. The remediation is not just patching; it is patching, regenerating server keys, reissuing certificates, revoking the old certificates, and re-establishing user-session credentials that may have been intercepted while the bug was exploitable. The operational work for this is substantial; I have spent the past week on it across the engagement portfolio.
The structural lesson is the one BULLRUN and the broader Snowden disclosures had already pointed at. The infrastructure that secures the public-internet TLS ecosystem is not invulnerable; it is, on the contrary, fragile in ways that the controls frameworks have been pretending it is not. OpenSSL is a single-implementation choke point — most major Linux distributions ship it, most major application servers link against it, most major TLS deployments terminate on it somewhere. A single vulnerability in OpenSSL therefore affects a substantial fraction of the public-internet TLS ecosystem simultaneously. This is not OpenSSL's fault — they are an under-resourced open-source project producing critical infrastructure — but it is a structural property of the current TLS deployment architecture that needs to be acknowledged and worked around.
For the engagement work, the past week has been a continuous chain of "is your TLS deployment vulnerable, what is the remediation timeline, when can we issue new certificates". The honest answers vary widely. Several clients had OpenSSL 1.0.1-affected versions at the public-facing layer; remediation is in flight. Several clients had OpenSSL behind front-end load-balancers running different TLS implementations (which terminated TLS at the load-balancer rather than at the application server) and were therefore not directly vulnerable; remediation is at the load-balancer layer. Several clients had OpenSSL on internal services that were not directly internet-facing and whose direct exposure is therefore lower; the remediation timeline is more relaxed but not zero, because the compromise of an internal CA private key has substantial knock-on consequences that need to be considered. The variance across the portfolio has produced the most operationally-engaging week of pen-test-style work I have run since the RSA SecurID compromise three years ago.
The certificate-reissuance angle is the part of the operational response that has produced the most work. Public-CA-issued certificates need to be revoked and reissued; the CAs have been overwhelmed with requests over the past week; the OCSP-and-CRL infrastructure is, on present evidence, struggling to keep up with the revocation volume. Several clients have ordered certificates that have not yet been delivered; several CAs are publicly acknowledging that their issuance pipelines are running days behind. The certificate-transparency infrastructure that has been emerging since the post-DigiNotar discussion is starting to be operationally useful here — clients who are concerned about whether their reissued certificates are being properly issued can monitor the CT logs — but the deployment is still partial. Ivan Ristić's SSL Labs test has been the practical tool of choice for verifying remediation; the tool has been quickly updated to test for Heartbleed specifically.
For the Hedgehog SOC, the detection-content additions for Heartbleed have been about catching exploitation attempts against client public-facing infrastructure. The Snort signatures have been published quickly and are running across all monitoring clients; we have seen approximately fifteen thousand exploitation attempts across the client base in the past week, the vast majority from automated scanners cataloguing vulnerable infrastructure rather than from targeted attacks. The targeted attacks we have observed have been characterised by sustained probing of particular endpoints and post-exploit credential-harvesting attempts; the patterns are not novel but the volume is.
The wider point I keep coming back to is the under-resourcing of the OpenSSL project. Two years of exploitable vulnerability in code that protects substantial portions of the global TLS infrastructure, found by chance through routine source review at Google and an external penetration-test by Codenomicon. The OpenSSL project has, at present, a small handful of part-time maintainers and minimal funding. The infrastructure that depends on them is operationally critical; the funding that supports them is operationally inadequate. There is going to be a serious conversation about how to fund critical open-source infrastructure following Heartbleed, and I will be following it; the Linux Foundation has been signalling some kind of cross-vendor consortium to address the issue, but the details are not yet public.
The next post is probably the continuing Heartbleed cleanup, the OpenSSL-funding conversation if it produces something concrete, or whatever else surfaces from the post-disclosure forensic work that several of my correspondents are doing on individual client incidents.