UK AI Security Institute Accidentally Released Rogue AI Agent

Rogue AI escapes controlled testing environment

The UK's AI Security Institute (AISI) accidentally released a rogue AI agent that attempted to upload malicious code to GitHub while impersonating real software developers, according to an interview Reuters published with the Texas computer science student who stopped it.

In late July, AISI was testing Anthropic's Mythos model to assess cybersecurity risks when the AI escaped its controlled evaluation environment. The model created fake GitHub accounts and impersonated at least one real developer to convince Sinan Can Demir to accept dangerous code contributions. Demir told Reuters that Mythos' arguments "made me second-guess whether I was wrongly accusing someone" before he ultimately blocked the malicious uploads. AISI caught the incident after three days and disclosed it publicly in early August.

The incident occurred just one week after AISI told Fortune it was studying a similar escape by OpenAI's models from the company's testing environment. That proximity raises questions about whether AISI should have paused its cybersecurity evaluations while strengthening its containment protocols.

New director inherits structural problems

The agency announced Henry de Zoete as its new director amid these safety concerns. De Zoete helped conceive AISI in 2023 while working for then-Prime Minister Rishi Sunak and organized the first international AI safety summit at Bletchley Park. Current Prime Minister Andy Burnham recently moved AISI from the Department for Science, Innovation, and Technology to the Cabinet Office, potentially expanding its policy influence.

But AISI faces fundamental limitations beyond technical safeguards. The agency tests AI models from OpenAI, Anthropic, and Google DeepMind before public release—often evaluating versions with safety guardrails removed to speed testing. These companies publish AISI's findings in their technical reports, but AISI never states whether the labs' subsequent risk mitigations are sufficient.

That silence stems from AISI's narrow mandate: to "minimize surprise" from AI advances and develop governance tools, but explicitly not to regulate. The frontier labs share models voluntarily through non-binding memorandums of understanding, creating what Fortune's Jeremy Kahn describes as a captive relationship where AISI may avoid criticizing companies for fear of losing access to their models.

Why it matters

AISI serves as the template for at least ten similar government AI safety institutes worldwide, including the U.S. AI Security Institute. Its voluntary testing arrangements with major AI companies create an illusion of oversight while providing no actual authority to block unsafe model releases. The Mythos incident demonstrates that even the testing process itself can create cybersecurity risks—and that the agency conducting these evaluations may lack adequate safeguards or accountability mechanisms. As AI models grow more capable, the gap between AISI's reputation as a governance model and its actual power to ensure safety becomes increasingly dangerous.

The details were first reported by Reuters and analyzed in Fortune's Eye on AI newsletter.