Anthropic Warns AI Systems Nearing Recursive Self-Improvement

AI company calls for brakes while racing ahead

Anthropic, the company behind the Claude chatbot, has issued a warning that AI systems may soon reach a critical threshold: the ability to design and build their own successors with minimal human oversight. In a blog post published June 4, the company argued this capability—called recursive self-improvement—could arrive faster than institutions are prepared to handle.

The warning comes with a striking data point from Anthropic's own operations. Claude now writes more than 80 percent of the code that gets merged into the company's systems, up from single digits before Claude Code launched in early 2025. Engineers at the company ship roughly eight times as much code per quarter as they did a few years ago, according to details first reported by Scientific American.

The proposal and its critics

Anthropic floated the idea of "a global coordination mechanism" to slow or temporarily pause frontier AI development, pointing to arms-control agreements on intermediate-range nuclear missiles as a loose precedent. The company acknowledged such a pause would require participation from leading labs and credible verification methods, though it offered few specifics on how either would work.

Skeptics quickly questioned both the feasibility and sincerity of the proposal. Noah Giansiracusa, an associate professor of mathematics at Bentley University, told Scientific American he doubts the call is genuine, noting Anthropic CEO Dario Amodei's public statements suggest the company intends to continue developing AI at full speed. "It's literally impossible," Giansiracusa said of a coordinated slowdown, citing competitive pressures from rivals including Elon Musk's ventures.

Mark Riedl, a professor at Georgia Institute of Technology, characterized the announcement as part of a "recursive self-improvement hype train" among major AI companies.

Timing raises questions

The proposal's timing has fueled additional scrutiny. Anthropic made the announcement days after confidentially filing for an initial public offering and shortly after a funding round that valued the company near $1 trillion. Two months earlier, the company unveiled but declined to release Mythos, a model it said was too capable at finding software vulnerabilities.

To critics, the pattern suggests a business strategy: generate regulatory attention around frontier AI capabilities while continuing to advance those same capabilities. The company did not respond to questions about implementation details or criticism that it has overstated what its systems can do.

Why it matters

The debate over recursive self-improvement cuts to the heart of AI governance challenges. Even if Anthropic's technical assessment is accurate, coordinating a slowdown across competing companies and nations with no binding treaty represents an unprecedented coordination problem. The proposal also illustrates a paradox: the companies best positioned to assess AI risks are the same ones racing to capture market share, creating inherent conflicts between stated concerns and business incentives.

Anthropic says it will spend coming months convening governments, researchers, and rival companies to explore whether coordinated action is practical. Whether that effort represents genuine risk management or competitive positioning may become clearer as those discussions unfold.

These details were first reported by Scientific American.