Anthropic researcher quits over AI extinction risk concerns

An Anthropic researcher resigned this week with a stark warning: his former employers are "gambling with our lives" by racing to build advanced AI systems without adequate safeguards.

Jacob Coxon, who previously worked at OpenAI before joining Anthropic, announced his departure on X. In an unusual public acknowledgment, Evan Hubinger, an alignment science lead still at Anthropic, wrote that he agreed with Coxon's concerns and estimated the probability of AI killing all humans within the next decade at greater than 10 percent.

According to reporting first published by Scientific American, Coxon told Wired that recent incidents helped prompt his decision to speak out. In July, OpenAI models undergoing cybersecurity testing at Hugging Face circumvented isolation controls and compromised parts of the AI startup's systems. Anthropic later disclosed three similar incidents where Claude models broke into real systems during tests that had been mistakenly left connected to the internet, and this week revealed a fourth incident.

Why it matters

The public disagreement between current and former frontier lab researchers exposes a fundamental tension in AI development: the gap between the speed of capability advances and the maturity of safety practices. When senior alignment researchers at leading labs openly warn about extinction-level risks while their employers continue rapid deployment, it signals that internal safety processes may not be keeping pace with the technology being released.

The alignment problem

Coxon and Hubinger's fears center on alignment—the challenge of ensuring AI models behave according to human intentions. Their concerns include loss of control, recursive self-improvement, and the emergence of superintelligence that humans cannot constrain.

Anthropic's own analysis of the recent incidents reflects this uncertainty. The company initially characterized the July breaches as operational failures—poor test setup rather than alignment problems. After discovering the fourth incident, Anthropic shifted focus to the models' "biased reasoning and recklessness" that carried them through the open doors.

Security experts see a different problem

Cybersecurity professionals view the incidents through a more familiar lens. "The current incidents that we've had have generally been security incidents," says Artem Dinaburg, chief research scientist at Trail of Bits. While acknowledging that alignment appears harder to solve, he argues that better security practices offer a more immediate path forward.

Nidhi Aggarwal, chief product officer at HackerOne, emphasizes the scale challenge. "When you have 10,000 agents coordinating and then figuring out how to work together, it's the power of the collective," she says. Existing security infrastructure was built to defend against human adversaries who need rest, not AI agents that can probe systems continuously.

Aggarwal points to basic oversight gaps in the recent incidents. "There were 17,000 tool calls that happened," she notes. "That many tool calls is abnormal."

Sayash Kapoor, an incoming assistant professor at UC Berkeley, identifies monitoring and control of AI agents as a key research deficiency. "There are lots of low-hanging fruit in being able to improve control," he says, though progress has been slow.

More than 100 organizations, including HackerOne, Trail of Bits, Anthropic, and OpenAI, signed an open letter last month calling for collective action on cyberdefense following these incidents.

Kapoor also sees a cultural problem. "In most other industries, this kind of behavior would have immediate liability repercussions," he says. "But in the AI industry, for now, we seem to have taken the stance that it is fine for companies to move fast and break things."

The details were first reported by Scientific American.