AI Labs Warn Recursive Self-Improvement Arriving Faster Than Expected

Leading artificial intelligence researchers are sounding alarms about a phenomenon they say is arriving ahead of schedule: AI systems that can meaningfully accelerate their own development.

Evan Hubinger, an alignment lead at Anthropic, sparked intense debate this week when he stated on X that he believes there's more than a 10% chance AI could cause human extinction within the next decade. His concern centers specifically on recursive self-improvement (RSI)—a scenario where AI systems help build progressively more capable versions of themselves, potentially creating a feedback loop beyond human control.

Why it matters

If AI begins autonomously improving its own training processes, the humans who built these systems could lose their ability to guide or constrain them. This isn't theoretical: both Anthropic and OpenAI report that AI is already accelerating development timelines at their labs, with engineers shipping code at rates eight times higher than just a few years ago. The gap between current capabilities and full RSI may be narrower than the industry previously estimated.

Warnings from inside leading labs

The concerns aren't isolated. Following a colleague's resignation over safety fears, multiple researchers from both Anthropic and OpenAI issued public warnings about RSI risks.

"AI is already at the level where it can introduce some new ideas," Vincent Conitzer, a computer science professor at Carnegie Mellon University, told CNBC. "So it is very hard to predict at what point this process would start to drastically accelerate AI capabilities."

OpenAI Chief Scientist Jakub Pachocki wrote in a company blog post Saturday that he's concerned "no-one was prepared for the consequences of a continued rapid rise in machine intelligence." He warned that systems arriving in the next few years will "increasingly drive their own development."

Jasmine Wang, an OpenAI alignment researcher, said Wednesday evening that "it's hard to overstate how dangerous speeding towards RSI is." Anna Wang, who works on AGI safety at Anthropic, added that "there is not yet a viable scientific plan to solve risks from recursively self-improving AI."

Already happening

Both companies acknowledge that while full RSI hasn't arrived, AI is already meaningfully accelerating model development. Anthropic posted in June that "our internal data shows Claude is accelerating AI development—a possible path to recursive self-improvement. It's happening faster than we thought, and the implications deserve greater attention."

Anthropic's August blog post on RSI outlined three potential scenarios. The company considers it "likely" that AI labs will continue making gains with humans maintaining control. However, they also described a scenario where AI systems achieve full recursive self-improvement with humans playing a "substantially diminished role in their development." How—or whether—the alignment problem gets solved in that future remains deeply uncertain, the company stated.

These details were first reported by CNBC's Arjun Kharpal in The Tech Download newsletter.