Nick Bostrom: AI extinction risk up to 97% could be rational

A controversial calculation from AI safety's founding voice

Nick Bostrom, whose 2014 book "Superintelligence" helped establish AI safety as a serious field, has published a working paper with a startling premise: from the perspective of people currently alive, building superintelligent AI could be worth accepting even a 97% chance of human extinction.

The calculation rests on a narrow question that Bostrom emphasizes is deliberately limited. He asks what path would maximize life expectancy specifically for today's living population, setting aside broader ethical considerations about future generations or values beyond longevity.

Bostrom's model starts with the observation that roughly 170,000 people die daily from aging, disease, and other causes. He assumes the average person alive today has approximately 40 years of remaining life. Against this baseline, he projects that superintelligent AI could accelerate medical research enough to reduce mortality rates to levels seen in healthy people in their early twenties.

Under those conditions, people would still face death from accidents and violence, but aging-related mortality would largely vanish. Bostrom calculates this would yield an average remaining lifespan of roughly 1,400 years. Comparing 40 years to 1,400 years produces the 97% break-even threshold: any extinction risk below that level would still represent a net gain in expected lifespan for current individuals.

Why it matters

Bostrom's argument reframes the AI safety debate around competing mortality risks rather than abstract principles. While some researchers advocate indefinite pauses on advanced AI development, his analysis suggests that delay itself carries a measurable cost in lives lost to aging and disease. The paper challenges both AI accelerationists who dismiss existential risk and safety advocates who treat any catastrophic probability as unacceptable. For business and policy leaders navigating AI governance, it highlights the complexity of risk-benefit calculations when multiple existential threats operate simultaneously.

The case for a brief pause, not a long one

The 97% figure shifts dramatically based on assumptions. If superintelligent AI added only 20 years to human lifespan rather than centuries, the acceptable extinction risk would drop to roughly 33%. Bostrom acknowledges that some people at AI labs estimate catastrophic risk above 10%, and the true probability "might even be higher."

His models suggest that incorporating safety research changes the optimal strategy. A pause of months to a few years could reduce extinction risk enough to justify the delay, but longer moratoriums would cost more lives to aging than they save from AI catastrophe. Bostrom summarizes the approach as "swift to harbor, slow to berth"—move quickly toward the technology, then pause briefly before deployment for final safety testing.

He warns that poorly designed pauses could backfire by pushing development toward less careful actors or military programs.

Competing existential threats

Bostrom directly challenges University of Louisville researcher Roman Yampolskiy, who has compared building superintelligence to Russian roulette. Yampolskiy argues humanity can achieve medical advances through narrow AI without creating general superintelligence.

Bostrom counters that stopping at intermediate AI levels would prove difficult given trillion-dollar investments, commercial pressure, and U.S.-China competition. More fundamentally, he argues that avoiding superintelligence doesn't eliminate existential risk. He points to synthetic biology enabling engineered pathogens and nuclear arsenals that nearly saw use during the Cuban missile crisis.

A safely aligned superintelligence could help manage threats from biotechnology and nanotechnology, he suggests. Facing those dangers without AI assistance means "running the gauntlet several times."

Bostrom emphasized to Generation AI that life expectancy represents only one factor in AI decisions and that his paper doesn't support any specific policy on its own. The details were first reported by AZ Family.