Follow Cyber Kendra on Google News! | WhatsApp | Telegram

Add as a preferred source on Google

LiteLLM Breach Exposed 434,000 CI/CD Pipelines, 2,500 Firms

Hudson Rock and CloudSEK say the LiteLLM supply chain attack exposed 434,000 CI/CD pipelines at 2,500+ companies.

LiteLLM 434,000 CI/CD pipelines

Two threat intelligence firms have published victim data from the March 2026 LiteLLM supply chain attack, and the scale is far beyond anything known when the malicious packages were pulled. CloudSEK counts more than 2,500 potentially exposed organisations and roughly 434,000 CI/CD pipelines, while Hudson Rock says it independently obtained the attackers' raw exfiltration archive and attributed 118,829 CI runner dumps to 2,488 corporate domains.

There is no CVE for this incident because nothing in LiteLLM itself was vulnerable. Attackers linked to the threat group TeamPCP published backdoored LiteLLM 1.82.7 and 1.82.8 to PyPI, where CloudSEK says the packages stayed live for roughly 40 minutes. 

Version 1.82.8 shipped a malicious .pth file — a Python startup hook that runs when the interpreter launches rather than when a library is imported — so the payload is executed on every machine where the package was merely installed, sidestepping the --ignore-scripts protection teams rely on to make installs safe.

How Did TeamPCP Get Into LiteLLM?

LiteLLM was never attacked directly. Its CI pipeline installed the Trivy vulnerability scanner unpinned from the system package manager, so when TeamPCP took over Trivy's release process, the poisoned scanner flowed into LiteLLM's build automatically, and that build published the malicious PyPI releases.

The opening was a single automation token that Trivy's maintainers rotated but did not fully revoke. CloudSEK says that left roughly a 20-day window in which the attacker force-pushed malicious code over the scanner's published version tags, meaning downstream builds pulling those tags received attacker-controlled code that still resolved and still looked legitimate. CloudSEK researchers summarised the cascade as "one un-revoked token, three tools deep".

Cyber Kendra covered the original LiteLLM poisoning on March 24, when Endor Labs first identified the malicious versions. The same campaign has since run through Checkmarx's KICS images, the Lightning AI PyPI package, and 42 TanStack npm packages.

What Did the Stealer Take?

On each compromised runner, the payload — tracked by Google as SANDCLOCK — escalated to root and swept SSH keys, AWS, GCP, and Azure credentials, Kubernetes service account tokens, .env files, and CI/CD secrets. CloudSEK reports it scraped secret values directly from /proc/<pid>/mem, including the values GitHub Actions attempts to mask in logs. Cloud keys were read straight from the instance metadata service and Kubernetes tokens from mounted service-account paths, using only the access each runner already carried.

For AI builds specifically, the stealer took LLM API keys and gateway configuration. Screenshots published by Hudson Rock show OpenAI, Anthropic, Gemini, OpenRouter, Fireworks, Groq, and Cerebras keys captured mid-execution — effectively the credentials to an organisation's entire model stack, plus its billing quota.

Collected data was encrypted with AES-256 under a hard-coded RSA-4096 key and shipped to a typosquatted domain. Where exfiltration failed, CloudSEK says the malware created a public repository inside the victim's own GitHub account and uploaded the stolen data there as a release asset, which means some organizations spent months publishing their own secrets without knowing it.

Do the 434,000 and 2,500 Figures Add Up?

The two reports were published a day apart and are being quoted interchangeably, but they do not measure the same things. Laid side by side, the discrepancies matter for anyone trying to size their own exposure.

Figure Hudson Rock CloudSEK Ars Technica
Archive size 153GB RAR Not stated 195TB
Files in the archive 433,909 Not stated Not stated
CI/CD pipelines 118,829 attributed runner dumps ~434,000 pipelines ~434,000 pipelines
Organisations 2,488 corporate domains 2,500+ companies 2,500+ organisations
Exposure window Audit from March 24, 2026 ~40 minutes 40 minutes

Two things stand out. First, the headline "434,000 CI/CD pipelines" figure is numerically almost identical to Hudson Rock's count of 433,909 files in the archive. Hudson Rock's own count of CI runner dumps could be attributed to an organisation of 118,829 — about 27 per cent of that total. 

The most likely reading is that the circulating pipeline figure counts records or files rather than distinct pipelines, and that roughly three-quarters of the dumps carried no usable organisational marker. Cyber Kendra has not independently verified either dataset, and neither firm has published a methodology reconciling the two counts.

Second, the archive size differs by roughly three orders of magnitude between Hudson Rock's own write-up (153GB) and Ars Technica's account of it (195TB). We have not been able to establish which is correct.

None of this makes the incident smaller. CloudSEK is explicit that its numbers describe reconstructed exposure and should not be read as proof that every listed organization was compromised. But a defender sizing their own blast radius should treat 434,000 as a count of leaked records, not as a count of breached pipelines, until either firm says otherwise.

Which Companies Are Affected?

CloudSEK's high-confidence list runs to dozens of names across technology, defence, banking, telecoms, manufacturing and logistics. It includes NVIDIA, Amazon Web Services, Samsung Electronics, Cisco Systems, Salesforce, ServiceNow, Siemens AG, S&P Global, Airbus U.S. Space & Defense, John Deere, Regeneron, London Stock Exchange Group, Thomson Reuters, FedEx, Munich Re, MediaTek, Volkswagen, Deloitte, Kroger, Thales, X Corp, Zscaler, Epic Games, Orange, HP, Philips, Vodafone, Carl Zeiss, Deutsche Bahn, NGINX, BT Group, Liebherr, Krungthai Bank and Roku.

The per-organisation counts are uneven in ways worth reading closely. X Corp shows 3,459 secrets across 1,153 runs. Orange shows 180 secrets but 5,642 runs. Volkswagen shows 2,242 runs and zero secrets. A high run count with no secrets suggests pipelines that executed the package without holding credentials the stealer could reach; a high secret count on a few runs suggests the opposite.

Attribution is also harder than the domain column implies. Hudson Rock describes one case where the committer email belonged to @siriusxm.com, but the environment dump's self-hosted GitLab and registry endpoints placed the breach inside AdsWizz, a SiriusXM subsidiary. Routing an alert to the wrong SOC on the strength of an email domain wastes the window that matters.

Hudson Rock's larger concern is the files with no marker at all. Many pipelines are configured generically, and the firm says those dumps hold "active database passwords, third-party API keys, and cloud credentials" with nothing tying them to a company. Absence from both firms' lists is not evidence of safety.

Are the Stolen Credentials Still Valid?

In at least one case, yes. After the Ars Technica report went live, Kevin Beaumont said a major US technology company told him it had rotated everything and the disclosure was a non-event. Its responsible disclosure policy permitted credential testing, so he tested them. "Almost every one worked," he wrote and filed a report.

That is the finding defenders should carry out of this story. These credentials date from March. Five months of assumed rotation did not close the exposure at one of the largest technology companies in the United States, which tracks with the FBI's July FLASH advisory (FLASH-20260702-01) warning that affiliated actors are likely to weaponise the harvested credentials long after the original intrusion.

What Should You Do If You Ran LiteLLM 1.82.7 or 1.82.8?

  • Rotate broadly, not narrowly. Rotating the LiteLLM or model-provider key alone is insufficient. Any credential readable by the affected process — in environment variables, process memory, on disk, injected into the job, or retrievable through an instance metadata service — should be treated as exposed until validated. That includes AWS, GCP and Azure IAM keys, Kubernetes service account tokens, and GitHub and GitLab personal access tokens.
  • Search build history, not just current lockfiles. Scheduled jobs, dependency resolvers, ephemeral runners, developer laptops and cached container layers can all hold the artifact long after PyPI removed it. Version 1.82.6 is the last confirmed clean release.
  • Hunt for persistence. Check site-packages for unauthorised .pth files and look for a systemd unit masquerading as a "System Telemetry Service". CloudSEK is also named tpcp-docs/docs-tpcp as a repository pattern worth hunting for.
  • Check your own GitHub org for repositories you did not create. The exfiltration fallback published stolen data as a release asset in the victim's account.
  • Pull logs from March 24, 2026, onward. Review CloudTrail, Kubernetes API audits, source control, package registry and cluster logs for token use from unfamiliar IPs, geographies, runners or user agents.
  • Check the exposure portals. Both firms have published domain lookup tools, including CloudSEK's. Note that both are gated behind vendor sign-up flows.

The structural lesson is the one Alon Gal, Hudson Rock's co-founder and CTO, put to Ars Technica: a single upstream breach now reaches thousands of companies at once, and a 40-minute dependency window produced hundreds of thousands of harvested environments. TeamPCP has since open-sourced the Shai-Hulud framework used across this campaign, which means the next actor to run this playbook will not need to build any of it.

Post a Comment