The Log4Shell response is in its tenth day. Three further CVEs have been disclosed against log4j during the response cycle — CVE-2021-45046 (a partial-mitigation bypass that affected the 2.15.0 release), CVE-2021-45105 (a denial-of-service and self-referential lookup issue), and CVE-2021-44832 (an additional remote-code-execution chain via JNDI in specific configurations) — each of which has required corresponding patch-upgrade cycles across the customer-portfolio Java estate. The cleanup posture is now to upgrade to log4j 2.17.1, which addresses the cumulative set of issues. The aggregate operational cost across the customer base has been substantial.
The customer-portfolio status as of this morning. Browne Jacobson's Java estate (smaller than most, primarily vendor-managed applications) is on 2.17.1 across all identified instances. Towry's trading-platform Java estate is on 2.17.1; several vendor-supplied Java applications in the wider firm-side estate are on 2.17.1 with the upgrade work having been accelerated. Northcott's Java estate is on 2.17.1 with one outstanding finding (a vendor-supplied operational-monitoring tool that the vendor has been slow to patch and that we are working with the customer to address through compensating controls). The manufacturer's substantial Java estate is approximately 95% on 2.17.1 by host count, with the long-tail being vendor-supplied applications where the patching is gated on vendor-side action; the customer-organisation hunt activity continues. The financial-services firm's Java estate is on 2.17.1. The retailer's Java estate is on 2.17.1.
The wider economy is in worse shape. The Cybersecurity and Infrastructure Security Agency has issued sustained guidance through the past week (CISA Apache Log4j Vulnerability Guidance hub) and has indicated that exploitation continues against unpatched estates at substantial scale. The customer-organisation conversations across the wider sector — the ones we are not directly engaged with but where colleagues at other firms are dealing with the same response — indicate that the patching cadence at organisations without strong SBOM discipline is materially slower than at organisations with the discipline, and the long-tail of unpatched instances will persist for years.
The structural-lesson conversation that is starting to settle. First, the software-bill-of-materials discipline is no longer an abstract concern but an operational necessity. The customer-organisation programme work for 2022 will substantially incorporate SBOM-and-software-composition-analysis tooling and process. Second, the Java ecosystem's dependency-tree depth is operationally challenging and the customer-organisation programmes need explicit visibility into what versions of what libraries are running in what applications. Third, the open-source-foundation funding question — log4j is maintained by a small volunteer team, the security-engineering effort against the codebase has been historically limited, and the sustained operational impact of vulnerabilities in this kind of widely-deployed open-source component is borne by the consumer organisations rather than by the maintainers. The 2022 conversation about open-source-foundation funding (the Open Source Security Foundation work, the various corporate-funding initiatives, the policy conversations about whether and how governments should fund critical open-source components) is going to be sharpened by Log4Shell as the worked example.
For the customer-portfolio strategic conversations through Q1 2022, the Log4Shell post-mortem is going to be a substantial agenda item at every customer-organisation board cycle. The operational lessons will be incorporated into the customer-organisation programme work. The cost of the response in operational terms will be reflected in the 2022 budget conversations across the portfolio. The Hedgehog operational cost has been substantial — approximately twelve full-team-equivalents of effort over the response period at a rough-cut estimate — and the financial-impact of the response on the company is going to be visible in the Q4 results and the early-2022 trajectory.
The team has performed exceptionally through the response. The personal-recognition and team-recognition pieces of the response are not for this blog but are for the internal records. The morale and engagement measures across the team have, on the past-week pulse-survey instrument, been higher than at any previous point of the year, which is the unexpected positive of the operational pressure — the team has met the moment, and the team's awareness of having met the moment is, in itself, a significant team-cohesion event.
I will return to this. The Log4Shell post-mortem will continue through Q1 and the structural lessons will inform customer-organisation programme work for years.