Four days after Stéphane Chazelas at Akamai disclosed CVE-2014-6271 — the bug in bash environment-variable parsing that allows arbitrary remote command execution in any service that passes user-controlled data into a bash environment — and the engagement work has been operationally similar to the Heartbleed week in April but with a worse threat model and a longer fix tail. Shellshock is, in some ways, more serious than Heartbleed. Heartbleed leaked memory; Shellshock executes code. The memory-leak version of catastrophe is bad; the remote-code-execution version is worse.
The technical chain is straightforward enough to write down in a paragraph. Bash supports exporting functions through environment variables, with a syntax that uses a function-definition string like () { ... } as the variable value. Bash's parser, on importing such a variable, evaluates the function definition. The parser, as it has been written since 1989, does not stop at the closing brace of the function definition; it continues parsing whatever follows the function in the variable value, and executes it. So a malicious environment variable value of () { :; }; rm -rf / (or whatever the attacker chooses) gets bash to evaluate the rm -rf / part as code at the moment the variable is imported. The bug is therefore a remote command execution vulnerability against any service that passes user-controllable data into bash environment variables. The list of such services is long and uncomfortable: CGI scripts (where HTTP headers become environment variables), DHCP clients (where server-provided strings become environment variables), various mail-processing systems, sudo invocations, ssh ForceCommand patterns. Anything that combines "user-controllable input" and "bash" has, on present evidence, been vulnerable since 1989.
The patch is the second part of the story. The original Red Hat patch for CVE-2014-6271, which shipped on the twenty-fourth, was incomplete. Within hours, Tavis Ormandy at Google had identified that the patch only addressed the most obvious form of the vulnerability and not several closely-related forms; CVE-2014-7169 was assigned for the residual exposure. Subsequent patches for CVE-2014-7186, CVE-2014-7187, CVE-2014-6277, and CVE-2014-6278 have followed through the past four days, with each new fix surfacing additional related issues. The bash parser is, on the evidence of the past week, considerably more fragile than anyone had previously appreciated. The patching work is not over.
For the engagement work, the past week has been continuous identification of bash-exposed services. The honest accounting across the engagement portfolio: most clients have substantial Linux estates, most of those run bash as the default shell, most of the bash-exposed services are internal rather than internet-facing but a meaningful proportion are public. The CGI-script exposure is the most pressing because it represents direct internet-facing remote-code-execution; we have spent the past week identifying CGI-using web applications across the portfolio and either patching or, where patching is not promptly possible, deploying WAF-level filtering against the obvious exploitation patterns. The DHCP-client exposure is the more concerning long-tail because it affects laptops and embedded devices in ways that the standard patch-management infrastructure does not always reach. The mail-processing exposure is variable; some clients have it, some do not. The cumulative scope is uncomfortable.
For the Hedgehog SOC, the detection-content additions have been around the obvious exploitation patterns — HTTP requests with () { :; } patterns in headers, particularly User-Agent and Referer; DHCP responses with the same patterns; outbound connections from previously-quiet hosts that follow the kind of post-exploitation pattern Shellshock enables. We have seen approximately twenty thousand exploitation attempts across the client base in the past four days, the overwhelming majority from automated scanners cataloguing vulnerable infrastructure rather than from targeted attacks. The targeted attacks we have observed have been focused on specific industries — there is one cluster of attempts against engagement clients in the legal sector that has the operational signature of intelligence-led targeting, though I cannot say more without identifying clients.
The wider point about under-appreciated infrastructure components is the same point I made about OpenSSL after Heartbleed. Bash is the default shell on most Linux distributions and on OS X. It has been operationally critical for twenty-five years. It has been shipping with this vulnerability for twenty-five years. The bash codebase is maintained by Chet Ramey at Case Western Reserve University, primarily as a side project. The under-resourcing of critical infrastructure is, on present evidence, the persistent structural problem of 2014. The Linux Foundation's Core Infrastructure Initiative has now added bash to its funding priorities; whether the funding actually translates into more rigorous code review and faster patch development for bash specifically remains to be seen.
The narrower question about how Chazelas found the bug is also worth recording. He was, on the public account, not specifically looking for it — he was working on something adjacent and noticed the parser behaviour. Twenty-five years of bash, with the bug exploitable across that entire window, and the discovery was essentially incidental. The implication for what else is in the codebases of critical infrastructure is uncomfortable. There are very probably more Heartbleed-and-Shellshock-class vulnerabilities sitting in widely-deployed open-source infrastructure waiting to be found by chance; the Core Infrastructure Initiative funding may help but the underlying problem of "we have substantially less code-review capacity than the size of the deployed codebase justifies" is the structural one.
The next post is probably either the continued Shellshock cleanup, the wave of incidental bash-related vulnerabilities that I expect to surface as researchers go through the codebase systematically, or whatever else surfaces. I will write again in mid-October when the dust settles, or sooner if something substantial breaks.