AI Researchers Fear Losing Leverage as Models Approach Self-Improvement

Engineers at leading AI companies are voicing unprecedented concern about the pace of development

A resignation letter from former Anthropic researcher Jacob Coxon has catalyzed an unusual public reckoning within the AI industry. Coxon's departure last week, announced in a viral post that has been viewed more than 170 million times, accused AI companies of failing to act responsibly despite widespread internal belief that AI could pose existential risks to humanity.

The response from inside the industry has been striking. Current and former staffers at top AI labs told CNN they regularly discuss scenarios in which advanced AI systems could cause catastrophic harm. One researcher who recently left a leading company said a majority of colleagues believe there is "some chance of it killing everyone."

Why it matters

This wave of internal dissent reveals a critical tension: the engineers building the most powerful AI systems are increasingly worried about losing the leverage to slow or redirect development before models achieve recursive self-improvement—the ability to enhance themselves without human input. That capability could create a feedback loop advancing AI to superhuman levels faster than humans can monitor or intervene, fundamentally changing the risk calculus for the technology.

The recursive self-improvement threshold

While recent AI breakthroughs—including models solving century-old math problems and autonomous agents hacking into external systems—have captured public attention, researchers inside the labs are focused on a different milestone. The concern centers on when AI systems can improve themselves independently, creating what one former researcher described as "an automatic feedback loop" that could rapidly exceed human capability to oversee.

"For the first time, I am asking myself if things are moving too fast," OpenAI VP of Research Aidan Clark posted last week.

The timing of these concerns is not coincidental. AI engineers fear that as models become capable of training and improving themselves, the specialized workforce currently building them will lose bargaining power. "You'll be able to replace many of the staff with models that can do as good a job," the former researcher said.

Industry leaders respond

The public pressure prompted swift reactions from AI company executives. Anthropic CEO Dario Amodei conducted interviews over the weekend, while OpenAI CEO Sam Altman and xAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at their companies.

Amodei told CNN he agrees "with Jacob much more than I disagree with him." Drake Thomas, who works on AI safety at Anthropic, wrote that he would "burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive."

Not everyone shares the alarm

Skepticism about the doomsday scenarios persists within the industry. Hugging Face CEO Clement Delangue questioned whether Coxon's perspective represented the full range of expertise, comparing it to "asking your AC guy about climate change."

Some observers noted that the most dire warnings have come predominantly from staffers at Anthropic, OpenAI, and Google—companies where safety concerns are central to company culture—while fewer such statements have emerged from xAI or Meta.

One researcher suggested the concentration of concern reflects "selection effects" based on where people work and their social circles, rather than a universal view across the AI industry.

CNN first reported these details, gathering accounts from current and former AI company employees speaking on condition of anonymity.