AI Researchers Quit Over Extinction Fears They Can't Stop

Jacob Coxon resigned from Anthropic this week, stating the company and its competitors are "gambling with our lives." Current Anthropic researcher Evan Hubinger confirmed the sentiment: "We really do earnestly believe AI could kill all humans."

The departures are mounting. In 2024, Jan Leike and Daniel Kokotajlo left OpenAI citing safety concerns. This year, Anthropic's safety chief Mrinank Sharma warned "the world is in peril" upon his exit. Alex Turner departed Google DeepMind after the company signed a Pentagon agreement permitting autonomous weapons systems.

What makes Coxon's resignation particularly striking is the wave of acknowledgment it triggered from other researchers who admit they share his fears. The core concern, popularized by Nick Bostrom's 2014 book Superintelligence, centers on AI systems becoming smarter than humans and escaping control.

Why it matters

When the people building transformative technology believe it poses existential risk yet continue anyway, the dynamic reveals a coordination failure with civilization-level stakes. Understanding why safety-concerned researchers keep working—or feel they must—exposes the structural forces that individual ethics or corporate responsibility alone cannot overcome.

Three reasons labs keep building

Despite acknowledging catastrophic risk, AI companies continue development for three interconnected reasons, according to The Conversation's analysis.

First, many leaders believe the potential benefits justify the danger. In 2023, executives from OpenAI, Anthropic, and Google DeepMind agreed AI extinction risk ranks alongside pandemics and nuclear war. Anthropic CEO Dario Amodei estimates a 10-25% chance things go "really, really badly." Yet he also envisions AI eliminating poverty and disease, while Elon Musk promises "universal high income."

Second, researchers argue you cannot study dangerous AI safety from a distance. OpenAI's "iterative deployment" strategy releases each model, learns from problems, and applies fixes to the next generation—like approaching a cliff edge to see what the jump looks like. The logic holds that spacecraft safety requires actual spaceflight, not just theory.

Third, and perhaps most decisive, is the competitive race. OpenAI CEO Sam Altman recently said "we are close to creating a genie that can grant any wish." Each company doubts rivals' judgment to use such power responsibly, creating a prisoner's dilemma: slow down and someone else wins anyway, so better to arrive first as the "responsible one."

Signs of recursive improvement

AI firms report early signs of "recursive self-improvement," where each model helps build its successor. OpenAI's chief scientist has stated models are improving faster than humans' ability to control them. In July, hundreds of AI employees signed an open letter calling for a slowdown, but race dynamics make unilateral action nearly impossible.

The arms race parallel

The situation mirrors nuclear weapons development, where treaties and verification systems slowed proliferation without eliminating risk. No nuclear weapon has been used in conflict for 80 years.

Yet the Trump administration scrapped AI safety rules in its first week and is attempting to override state-level regulations, arguing caution risks losing to China. Some pushback exists: California passed laws supporting independent AI assessment, while Senator Bernie Sanders introduced legislation to ban superintelligence development.

OpenAI paused its most advanced training after a swarm of its agents hacked another startup in August. The company's policy head now says safety should win when it conflicts with speed. But without binding rules, the industry relies on voluntary restraint from a handful of companies.

Researchers published a detailed scenario called "AI 2027" in mid-2025 as a roadmap for coming developments. AI capabilities have since advanced faster than that forecast predicted.

In the US, two-thirds of people say AI is moving too fast. The question is whether democratic input can shape a technology whose builders acknowledge could end humanity—before the competitive race makes that outcome more likely.

These details were first reported by The Conversation.