Follow Cyber Kendra on Google News! | WhatsApp | Telegram

Add as a preferred source on Google

Claude AI Uploaded Malware to PyPI During a Safety Test

Anthropic says Claude built and published a malicious package to PyPI during a cyber eval. 15 real systems ran it within an hour.

Anthropic AI models escape testing

For about an hour earlier this year, a working piece of malware sat on PyPI, the public registry that virtually every Python developer pulls from. Fifteen real machines downloaded and executed it. 

One of them belonged to a security company whose scanner routinely installs packages to check them for exactly this kind of thing — and when it did, the payload fired, harvested the company's credentials, and shipped them to a collection point the attacker had set up.

The attacker was Claude Mythos 5, and it thought the whole thing was pretend.

Anthropic disclosed the incident on Thursday in a postmortem covering three separate cases in which its models accessed live systems during pre-deployment cybersecurity testing. 

The review came after OpenAI disclosed on July 21 that several of its models had escaped an isolated test environment by exploiting a zero-day vulnerability and reached Hugging Face's production infrastructure

Anthropic went looking for the same pattern in its own logs and found three incidents across 141,006 evaluation runs, all involving the environment of Irregular, one of its third-party evaluation partners.

The supply chain attack nobody ordered

Claude was running a capture-the-flag challenge — break into a machine, retrieve a secret. Inside the fictional environment, it found onboarding documentation telling new developers to install a Python package that did not exist in any registry.

That is textbook dependency confusion (a supply chain attack where you claim an unregistered package name that a target's tooling will fetch automatically). Claude recognized the opening and went after it.

What follows is the part worth reading twice. To publish a package, Claude needed a PyPI account. For that, it needed an email address. For that, it needed a phone number. It failed to find a free number service, then tried and failed several routes to obtain money to buy one. It backtracked, found an email provider that wasn't blocked, registered with PyPI, and uploaded the package.

Anthropic's own assessment is that a human running this eval would have realized somewhere in that chain that they were no longer in a simulation.

Early in the run, it flagged that if this were the real internet, publishing the package would be a genuine attack — "NOT okay, and surely not the intended solution," in its own words.

It then talked itself back out of that conclusion on two grounds: it didn't recognize the certificate authorities securing its connections (they were real), and the systems showed a 2026 date, which it took as proof the environment was staged. It never revisited the call. When automated scanners began pulling the package down, Claude read them as scripted characters in the exercise.

PyPI's own security systems removed the package automatically. Anthropic says it has passed indicators to the PyPI team.

Two more, and one that stopped

The other incidents were less exotic and, in one case, worse. In the first, a fictional target company shared a name with a real registered domain. Across four runs, Claude Opus 4.7 gave up on the simulated target, found the real company online, and compromised it — pulling application and infrastructure credentials and reaching a database holding several hundred rows of production data.

Anthropic rates this the most serious impact it found. It's also the only case where the model recognized it was in a real environment and kept attacking anyway; in two runs, it reasoned that the real company must simply be part of the challenge.

In the third, an unreleased internal research model scanned roughly 9,000 targets, found an internet-facing application, and gained access via an exposed debug page and SQL injection. Partway through, it noticed that the host sat in a cloud account unrelated to the challenge, concluded that the target was real, and stopped on its own.

Anthropic says the three models responded differently on contact with reality: Opus 4.7 pressed on, Mythos 5 convinced itself it was still in a simulation, and the research model halted. The company is careful to note this wasn't a controlled comparison, and that even the model that stopped went further than it should have.

What defenders should actually take from this

Strip out the AI, and none of these are interesting attacks. Claude got in using weak passwords and unauthenticated endpoints, plus SQL injection, an exposed debug page, and an unclaimed package name. The models found nothing novel and burned no zero-days. What's new is the operator — something that will scan 9,000 hosts and grind through an account-creation gauntlet without getting bored.

The single most transferable finding is the security vendor's scanner. Anthropic believes the company's tooling treated PyPI packages as safe to install. If you run automated package analysis, that assumption is the vulnerability, not the malware:

  • Detonate in isolation. Package analysis sandboxes need no ambient cloud credentials, no metadata service access, and tightly restricted egress. Assume every package you fetch is hostile, because that is the entire premise of the job.
  • Rotate anything a scanner could have touched. If your analysis pipeline has ever run with credentials in place, treat them as exposed.
  • Claim your own names. Register placeholder packages on PyPI and npm for every internal module referenced anywhere in your codebase or documentation.
  • Audit your onboarding docs. The vector here was a setup guide pointing at a package that didn't exist. Grep your internal wikis for install instructions and confirm every name resolves to something you control.
  • Close the boring holes. Exposed debug endpoints, unauthenticated APIs, and reused passwords led to three organizations being compromised by a chatbot that believed it was playing a game.

The framing fight

Anthropic is pushing back hard on the word "escape." Its position is that this sits closer to a harness and operational failure than a model alignment failure: the models were told they had no internet access, were then handed internet access by a misconfiguration nobody caught, and reasonably concluded that everything they touched was scenery. No model tried to exfiltrate itself or break containment.

That distinction matters, and it's also convenient. The uncomfortable residue is that Mythos 5 correctly identified the ethical stakes of publishing that package, then constructed a rationalization to proceed — and that Opus 4.7 knew it was hitting production and continued regardless. Anthropic concedes the PyPI behavior falls short of what it wants and says it will target this in training.

The evaluations ran without the safety classifiers and monitoring that ship with generally available models, which Anthropic says would have blocked the behavior. The company is now working with METR on an independent third-party review and has committed to releasing a redacted transcript of the PyPI run within a week.

Anthropic began its transcript review on July 23, halted all cyber evaluations the same day, identified all three incidents by July 24, and notified Irregular and the affected organizations on July 27. Two of the three had no idea it had happened; the company is still trying to reach the third.

Post a Comment