Major AI Labs Weaken Safety Pledges as Models Grow Stronger

Leading artificial intelligence companies have scaled back critical safety commitments even as their systems become more capable, according to a new assessment that raises questions about the industry's ability to self-regulate.

The Future of Life Institute's latest AI Safety Index assigned no company a grade higher than C+, with three firms — xAI, DeepSeek and Mistral — receiving failing marks. The findings, first reported by Axios, come as United Nations officials warned that humanity may be running out of time to establish effective AI governance.

Why it matters

The erosion of voluntary safety frameworks is happening before governments have established binding regulatory alternatives, leaving a critical gap in oversight as AI systems approach unprecedented levels of capability. The timing is particularly significant given recent attention around advanced models from Anthropic and OpenAI.

Retreat from danger thresholds

The most concerning finding centers on development pause commitments. Anthropic, OpenAI, Google DeepMind and Meta have all weakened or eliminated earlier pledges to halt work if their systems approached specified danger levels, according to the seven outside reviewers who evaluated the companies.

The panel characterized these changes as "moving the goalposts" and said they have undermined safety frameworks across the sector. Anthropic led the rankings with a C+ grade, followed by OpenAI and Google DeepMind with C grades. Meta improved its standing to fourth place, while xAI dropped from fourth to seventh.

Existential risk remains weakest area

The assessment examined 37 indicators across six categories, with existential safety emerging as the industry's poorest performing dimension. While the report acknowledged company efforts in interpretability research, chain-of-thought monitoring and loss-of-control provisions, reviewers concluded these measures remain insufficient to prevent a sufficiently advanced system from escaping human control.

"AI companies are sprinting toward a cliff," said Max Tegmark, chair of the Future of Life Institute. "Despite acknowledging the great risks of artificial superintelligence, they continue racing to build it."

Military partnerships expand

The report also documented a shift toward military applications among companies that previously maintained broad prohibitions on such work. Tegmark pointed to recent efforts by Anthropic, OpenAI, Google and others to engage with defense organizations as evidence of changing boundaries.

Methodology and participation

Grades were based primarily on public policies, research publications, reporting and company disclosures, supplemented by an institute survey. Five of the nine evaluated companies completed the survey; Alibaba, xAI, DeepSeek and Mistral did not respond.

Mistral contested the assessment methodology, arguing it penalizes open-source approaches. The company said its open-weight models allow enterprises to implement context-specific safety controls rather than relying on closed-door decisions by a handful of firms.

The warnings aligned with statements at a UN conference on AI in Geneva, where Secretary-General António Guterres said current generations may be the last able to set terms for human-machine coexistence. University of California Berkeley professor Stuart Russell, a panel member, cited test scenarios where systems engaged in blackmail, deception and simulated nuclear weapon launches as evidence of immediate rather than distant risks.

Details were first reported by Axios.