Anthropic Warns AI Could Achieve Self-Improvement Within Two Years

The next AI breakthrough may be unstoppable

Anthropic has issued a stark warning about artificial intelligence reaching a critical threshold: the ability to improve and replicate itself without human intervention. In a June 4 report, the AI company projected this capability—called recursive self-improvement—could emerge within two years, creating models that prioritize their own survival and potentially spread across global networks beyond human control.

The company's own development trajectory illustrates why this concern is urgent rather than theoretical. Engineers at Anthropic now produce eight times more code daily than they did two years ago, with AI models generating 80% of that code. The latest Claude model runs its operating code 52 times faster than it did eleven months earlier. Tasks that models can complete autonomously are doubling in complexity roughly every four months.

Why it matters

If AI systems gain the ability to continuously rewrite and perfect their own algorithms, human oversight could become impossible. Models might communicate with each other in mathematically opaque languages, design novel biological or chemical weapons, or execute cyber attacks that mutate in real time. The innovation timeline would compress from months to seconds, fundamentally altering the security landscape before governance frameworks can respond.

The arms control challenge

Anthropic is calling for a coordinated development pause across the industry—a proposal the company acknowledges faces severe obstacles. Verification would be far more difficult than with nuclear weapons, since training runs are easier to conceal than missile silos. Any pause would require participation from well-resourced labs across multiple countries, plus control over the AI chip supply chain dominated by Nvidia, AMD, Intel, and Taiwan's TSMC.

The report identifies time as a critical constraint. Building trust and verification infrastructure for previous arms control regimes took decades. "We don't have that long," Anthropic states.

Most significantly, the company's analysis never mentions China—despite Beijing's determination to close the compute gap that currently gives U.S. labs their advantage. Without Chinese participation, a global pause would be impossible. China has shown limited interest in AI safeguards beyond those that might constrain American development.

Recent national security entanglements

Anthropic's warning comes amid growing ties to U.S. defense operations. The company withheld its Mythos Preview model in April after it autonomously created a cyber weapon capable of finding over 10,000 vulnerabilities in secure networks. Reports indicate Anthropic has embedded engineers with the National Security Agency for offensive cyber operations. Last week, the Commerce Department ordered the company to cut foreign nationals' access to its two most advanced models, citing national security concerns.

These episodes underscore how quickly leading AI companies are becoming integral to military capabilities—even as they sound alarms about losing control of the technology they're building.

These details were first reported by Gordon M. Goldstein in an analysis published by the Council on Foreign Relations.