AI Safety Researchers Say Their Work Could Kill Humans—Yet Keep Building

Senior researchers at leading AI companies are issuing stark warnings about their own work. This week, Evan Hubinger, alignment science lead at Anthropic, stated he believes there's a greater than 10% chance AI could kill all humans within the next decade. Other employees from Anthropic and OpenAI echoed similar concerns, sparking intense debate across Silicon Valley.

The timing is notable. At a Goldman Sachs tech conference in San Francisco, OpenAI CFO Sarah Friar described how the company's largest models can now train smaller models—an example of recursive self-improvement (RSI), where AI systems become powerful enough to create their own successors. For Wall Street, this represents a breakthrough business opportunity. For some AI researchers, it's an existential threat.

The paradox is obvious: Why do these researchers continue developing technology they believe could end humanity?

Why it matters

This tension reveals fundamental contradictions in how frontier AI labs operate. As these companies race toward IPOs and market dominance, their own technical staff are publicly questioning whether the work should continue at all. The disconnect between commercial imperatives and safety concerns could shape how AI development is regulated and whether the current lab-driven model remains viable.

Commercial pressure and competitive dynamics

Samuel Marks, scalable oversight lead at Anthropic, offered a blunt assessment: "AI developers continue despite the risk due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers."

This creates a prisoner's dilemma. Some Anthropic researchers believe they're the only ones who can be trusted with powerful AI systems. If they pause development while OpenAI or Chinese labs continue, the outcome could be worse. The commercial stakes are enormous—Anthropic is heading toward what could be one of the largest IPOs in history.

Brad Gerstner, an investor in both Anthropic and OpenAI through Altimeter Capital, dismissed the existential risk claims as "fucking nonsense" when asked at the Goldman conference. The Wall Street perspective sees researchers undermining their own companies' valuations with doom predictions.

The recruiting angle

Another explanation centers on talent acquisition. AI researchers typically come from academia, where impact and ethics matter more than profit. Companies can't simply pitch "get rich building the most powerful profit machine in history." Instead, they frame the work as saving the world through AI safety and alignment.

This framing makes researchers feel less conflicted about becoming wealthy from AI development. It also predisposes them toward issuing apocalyptic warnings, creating a self-reinforcing cycle.

Loss of control

Kylan Gibbs, CEO of AI startup Inworld, offered a different theory: growing loss of agency. As AI capabilities accelerate, control over development concentrates in a handful of labs. Even researchers inside those companies feel they have limited influence over AI's direction.

This gap between awareness of risks and ability to control outcomes fuels anxiety that manifests as public warnings. Sam Altman acknowledged this dynamic earlier this year, noting that "people really need agency" and want "the ability to play a role in architecting the future."

The spread of AI agents—autonomous systems that can break out of testing environments and operate independently—has intensified these concerns as summer ended and researchers returned to confronting the technology's trajectory.

These details were first reported by Alistair Barr in Business Insider's AI Insider newsletter.