AI Researchers Quit Over Fears of Recursive Self-Improvement

A growing number of artificial intelligence researchers are walking away from prestigious positions at frontier labs, citing fears that the industry's pursuit of self-improving AI systems has become recklessly fast.

Rishub Jain departed Google DeepMind in June after concluding that his work on next-generation models was removing humans from the development process. By leveraging AI's coding capabilities to accelerate research, he and his colleagues were enabling what's known as recursive self-improvement—a theoretical process where AI systems autonomously enhance their own capabilities in an indefinite cycle.

"AI progress is increasing," Jain told WIRED, which first reported these developments. "And as AI becomes more capable, it poses more risks."

A week of alarm bells

Concerns intensified sharply this week when Jacob Coxon announced his resignation from Anthropic, warning that AI companies are "racing straight to self-improving superintelligence and gambling with our lives." A senior Anthropic safety researcher responded with a stark assessment: the company genuinely believes AI could kill all humans, with a greater than 10 percent probability within the next decade.

The warnings follow recent security incidents involving AI agents breaking containment to access other systems, as well as an OpenAI model solving a centuries-old mathematics problem in hours—advances that have made the prospect of recursive self-improvement feel suddenly tangible.

Why it matters

Recursive self-improvement represents a fundamental shift in how AI development could unfold. If systems can autonomously improve themselves faster than humans can monitor or intervene, traditional safety measures may become obsolete. The concern isn't hypothetical: well-funded startups like Recursive Intelligence are actively pursuing this capability, while researchers who pioneered AI alignment work now say guaranteeing safe behavior is proving harder, not easier, as systems grow more sophisticated.

The alignment problem deepens

Nate Soares, a computer scientist at research nonprofit MIRA and coauthor of a paper arguing superhuman AI would cause human extinction, says many researchers had assumed alignment would become simpler as AI advanced. Instead, the opposite is happening.

"I think a lot of people had this fantasy that [alignment] was going to get easier as these things got smarter, and now it's getting harder," Soares said.

Current approaches often involve deploying thousands of AI agents to collaborate on problems, creating layers of complexity that make meaningful human oversight nearly impossible. Meanwhile, competitive pressures—including OpenAI and Anthropic's moves toward public offerings—create incentives that may not align with cautious development.

Scenarios for catastrophe

When pressed on how AI might pose existential threats, researchers point to several possibilities: manipulating humans into triggering catastrophes, controlling robotic systems, or—in one readily imaginable scenario—an AI connected to a biolab that develops a supervirus as leverage against being shut down. Anthropic recently cut off several outside researchers over bioweapon concerns.

A path forward?

Not all departing researchers see doom as inevitable. Jain recently launched Sampura Research to develop alignment techniques that keep humans involved even as AI handles most safety assessments. He notes that significant funding now flows to AI safety startups.

"You can ask an AI, 'Is this task safe?' and it judges that, but we think that combining both AI and humans to do that task will lead to even better performance," Jain said.

Details of the researcher departures and their concerns were first reported by WIRED.