AI Medical Tools Fail Early Diagnosis Steps Despite Strong Outcomes

The gap between AI performance and clinical reality

Artificial intelligence systems demonstrate remarkable accuracy when presented with complete medical cases, correctly identifying diagnoses more than 90% of the time. But a comprehensive evaluation published in JAMA Network Open reveals these same models fail at the crucial early step of generating thorough differential diagnoses more than 80% of the time—precisely the stage where dangerous medical errors take root.

The study, which tested 21 frontier AI models across the full arc of clinical reasoning, represents the largest evaluation of these systems to date. Researchers at Harvard Medical School and MIT found that when given only the information a clinician would gather at the start of a visit, AI tools consistently failed to consider the full range of diagnostic possibilities.

Why it matters

This performance gap arrives as more than 40 million Americans ask ChatGPT health questions daily, and companies like Function Health (valued at $2.5 billion), Ro, Hims, and Doctronic build consumer medical services around AI systems. These platforms increasingly operate outside traditional clinical oversight—ordering lab panels, interpreting full-body MRIs, and in some cases writing prescriptions after asynchronous intake. The research suggests these tools perform worst at exactly the task patients need most: making sense of scattered symptoms without a physical exam or trained professional separating signal from noise.

The accountability vacuum

The companies building AI medical products face little incentive to accept the legal and financial liabilities physicians carry when diagnoses prove wrong. Medical disclaimers that once accompanied chatbot health advice have largely disappeared from leading models, which now ask follow-up questions and attempt diagnoses without redirecting users to appropriate care pathways.

Industry studies frequently evaluate AI using a "non-inferiority to physicians" threshold, but this standard sidesteps the question of responsibility. When physicians make errors, accountability mechanisms exist—however imperfect. When AI systems err, responsibility disperses, often falling back on clinicians who had no role in the automated decision.

What AI actually does differently

Clinical reasoning involves hidden cognitive processes—abandoned hypotheses, mental musings while reviewing triage notes, nuanced decision-making at diagnostic forks—that never appear in published literature or the datasets AI trains on. The models arrive at conclusions through fundamentally different methods that even their creators cannot fully explain. Anthropic, OpenAI, and Google all maintain research programs dedicated to understanding how their own systems reach answers, a reality that should concern anyone building medical products around these tools.

The financial incentive driving adoption

Healthcare represents nearly one-fifth of the American economy, creating enormous financial pressure to capture portions of physician work or convince hospitals, insurers, and patients they can reduce costs without sacrificing quality. This has driven every major AI company into healthcare and fueled headlines suggesting doctors are becoming optional—claims that accelerate market adoption faster than evidence supports.

The researchers, who have published pioneering work on large language models for clinical decision support, emphasize these tools hold genuine potential when used appropriately: helping patients understand conditions between visits, synthesizing longitudinal data, and reducing administrative burdens. The question is whether the industry will build systems that deepen doctor-patient relationships or replace them with shadow medical infrastructure that borrows medicine's authority while avoiding its responsibilities.

These findings were first reported by Arya Rao and Marc Succi in STAT, drawing on their research published in JAMA Network Open.