Codecov, the code-coverage-reporting service used in many continuous-integration pipelines, disclosed yesterday that the company's Bash Uploader script had been modified by an unauthorised actor on approximately the 31st of January 2021, and that the modified script had been served to customers from that date through approximately the 1st of April when the modification was discovered (Codecov security update, April 15). The modified script, on Codecov's analysis, included code that exfiltrated environment variables from the CI environment to an external server controlled by the actor. Environment variables in CI environments routinely contain credentials — cloud-platform API keys, source-code-repository tokens, deployment-platform credentials, third-party-service API keys — and the exposure of those credentials to an external party for two months has implications for the customer organisations whose CI pipelines used the affected uploader.
The mechanism is, in shape, the supply-chain-attack-into-CI pattern that the security-research community has been writing about for several years. The Codecov case is one of the more operationally consequential public examples of the pattern. The compromised vector — the script that customers download and execute as part of their CI workflow — has direct execution privileges in the customer's CI environment, which has, by typical configuration, access to substantial credential surface. The blast radius of the compromise scales with Codecov's customer base, which is large and includes substantial enterprise customers.
The customer-organisation response question is what to do about the credential exposure. The conservative posture — rotate every credential that may have been exposed in the affected CI environment during the two-month exposure window — is operationally substantial. Customer organisations using Codecov have, since yesterday, been working through credential-rotation cycles across cloud-platform, source-code-repository, and various third-party-service API credentials. The cost of this work is substantial; the cost of not doing it (continued credential exposure to a state or criminal actor with two months of accumulated harvested credentials) is larger.
For the customer-portfolio response, the audit found three customer organisations using Codecov in their CI pipelines — the manufacturer's product-engineering subsidiary, the financial-services firm's customer-platform engineering team, and the retailer's e-commerce engineering team. All three have been notified and are running credential-rotation cycles. The aggregate work across the three is the dominant Q2 SOC engagement for the customer-organisation security teams.
The wider strategic point — and this is going into the customer briefings for the next several months — is that the CI/CD pipeline is, in 2021, a substantial target surface for supply-chain attacks. The development-platform credentials that CI/CD environments process produce, on compromise, downstream effects across the customer organisation's cloud-deployment estate, source-code repository, and third-party-service relationships. The defensive disciplines — credential-scoping (CI credentials should be narrowly-scoped and short-lived), secrets management (centralised secret stores rather than environment-variable-scattered credentials), CI-environment integrity verification (signed and pinned third-party-script dependencies), and CI-environment monitoring (logging and detection on what CI processes actually do) — are well-known but unevenly deployed.
The customer-portfolio conversations this quarter are going to invest in the CI/CD security discipline at customer organisations where it has been an underdeveloped area. The pen-testing engagement queue is showing the predictable uptick in customer demand for CI/CD-focused engagements. The product-roadmap conversation for EmilyAI is, separately, considering whether the SOC-detection-content that the model uses can be extended into CI/CD-pipeline-event ingestion as a future capability. The operational-detection-against-CI-anomaly question is one we have been discussing with the engineering team through the autumn; the Codecov case will accelerate the conversation.
I will return to this as the post-Codecov customer-organisation work develops.