AI Extinction Warnings Lack Empirical Basis, ITIF Argues

Recent warnings from AI researchers about catastrophic risks have reignited debate over whether frontier AI development should slow down. Earlier this month, Anthropic researcher Jacob Coxon resigned and warned that AI builders "earnestly believe that it could kill us all by the end of the decade." His colleague Evan Hubinger put a number on it: greater than 10 percent chance of human extinction within ten years.

But according to Daniel Castro, president of the Information Technology and Innovation Foundation, these probability estimates have no empirical foundation. In an analysis published by ITIF's Center for Data Innovation, Castro argues that while AI presents measurable risks, extinction scenarios rest on unsupported assumptions.

Why it matters

As policymakers worldwide consider AI regulation, distinguishing between evidence-based risks and speculative catastrophes will determine whether governance frameworks enable innovation while managing genuine threats or impose costly restrictions based on unquantifiable fears. The debate has immediate implications for research funding, international competitiveness, and the pace at which beneficial AI applications reach patients, scientists, and businesses.

The Evidence Gap

No AI system has ever become an autonomous, self-improving agent with independent goals of eliminating humans, Castro notes. There is no historical record to calculate probabilities from, no empirical model translating today's capabilities into extinction risk.

The extinction scenario requires a long chain of assumptions: an AI system must become vastly more capable, develop conflicting objectives, gain autonomy and resources, evade human detection, and overcome technical and physical constraints. While researchers have demonstrated pieces of this chain in controlled settings, Castro writes, little evidence suggests the entire sequence will occur in the real world.

Measurable Risks Deserve Attention

AI does present real, documentable risks. Systems can facilitate fraud, cyberattacks, and disinformation. Researchers have documented cases where AI agents behave unexpectedly or circumvent restrictions. These warrant serious investment in security, testing, monitoring, and accountability.

Castro points to frontier models' demonstrated ability to identify and exploit vulnerabilities in digital systems as justification for a coordinated national AI cyber defense initiative—a concrete response to a measurable threat.

The Slowdown Problem

Calls to slow AI development face practical challenges. What exactly should slow down? Which models, training runs, or companies? For how long? What measurable safety conditions would allow resumption?

AI development lacks a single on-off switch. Researchers can independently adjust model architecture, training methods, compute levels, deployment practices, and safety testing. They're pursuing fundamentally different approaches, from large language models to systems building persistent representations of the physical world.

The international dimension complicates matters further. China has already rejected U.S. slowdown proposals, accusing American tech leaders of using safety concerns to constrain Chinese progress. Other countries continue heavy AI investment. A global slowdown would require countries with divergent interests and security priorities to agree on common limits and comply with them.

Capability and Safety Can Align

More capable AI could increase both the ability to perform harmful actions and the ability to recognize dangerous behavior, follow constraints, detect attacks, and prevent harm. Capability and safety can move in different directions, Castro argues. The goal should be understanding that relationship and making increasingly capable systems increasingly reliable.

Continued progress might even enhance safety. More capable AI could provide better tools for automated red-teaming, vulnerability discovery, interpretability, cybersecurity, and testing safeguards. Limiting capability could limit both risks and the tools available to manage them.

A Practical Path Forward

Castro proposes a safety agenda addressing evidence-backed risks while preparing for more serious threats as capabilities develop: establish clear liability for demonstrably harmful conduct, invest in testing for high-risk capabilities, strengthen cybersecurity and model containment, improve independent evaluation where meaningful benchmarks exist, and expand alignment and interpretability research.

Policymakers should demand clear evidentiary basis for catastrophic risk claims, specify which risks proposed interventions address, and weigh the costs of delaying beneficial technology against the risks of moving too quickly, he concludes.

These details were first reported by the Center for Data Innovation at ITIF.