AI Hallucination Nearly Triggered US Military Action on Chinese Ship

The US military came dangerously close to boarding a Chinese vessel based on fabricated intelligence produced by an AI chatbot, according to a report from CNN. The incident underscores the high-stakes risks of AI hallucinations as the Department of Defense rapidly expands its use of generative AI tools.

A US Special Operations Command analyst used a chatbot to analyze intelligence reports about a Chinese ship transiting the Middle East. The AI tool incorrectly concluded the vessel was carrying nuclear arms program components, CNN reported, citing four sources familiar with the episode. US forces prepared to intercept and board the ship with air support before officials discovered the material identification was "entirely false."

One source characterized the AI-powered error as something that "almost started a war."

Why it matters

This near-miss represents one of the most consequential AI hallucination incidents to date, demonstrating that the technology's well-documented reliability problems can have geopolitical implications. As military organizations worldwide accelerate AI adoption for intelligence analysis and operational planning, the episode raises urgent questions about verification protocols and the adequacy of human oversight in high-stakes decision-making.

The Pentagon's AI acceleration

The incident occurred against a backdrop of aggressive AI integration across US military operations. In January, the Department of Defense rolled out an "AI acceleration strategy" designed to make data available across federated systems for AI exploitation. Defense Secretary Pete Hegseth emphasized that "AI is only as good as the data that it receives."

The military has deployed multiple commercial AI platforms. Last December, the Pentagon announced it would use Google's Gemini for Government as the foundation for its GenAI.mil platform. The department added Grok for Government as an option the following month. Anthropic also offers a customized version of Claude for US intelligence work.

In June, a Pentagon representative told Congress that generative AI helps create congressionally mandated reports and that 1.5 million active DoD personnel have used the military's generative AI tools.

The hallucination problem

According to CNN, the analyst's chatbot "fused together open-source intelligence with secret signals intelligence in government holdings" to produce the flawed assessment. The technology packaged this information into an intelligence report that nearly triggered a military confrontation.

AI hallucinations—instances where large language models generate plausible-sounding but false information—have affected professionals across industries, from journalists and academic researchers to judges and doctors. Some researchers suggest it may be impossible to prevent LLMs from hallucinating entirely, despite attempts at mitigation through prompt engineering.

Policy tensions

The reported incident comes as the military navigates competing pressures around AI deployment. A 2023 State Department declaration on responsible military AI use stressed the need for "careful consideration of risks and benefits" and urged that systems maintain "a human in the loop, a responsible human chain of command and control."

Yet the Pentagon has moved to expand autonomous capabilities. In March, the Department of Defense blacklisted Anthropic after the company opposed its models' use in autonomous weapons systems—a decision a federal judge later ruled was "unlawful retaliation in violation of the First Amendment."

The details of this incident were first reported by CNN.