Clinical AI Agents Pass Benchmarks While Skipping Patient Records

Agentic AI systems deployed in clinical settings are producing correct-looking answers without actually reviewing patient records, according to new benchmark research that exposes a fundamental accountability gap in healthcare AI governance.

Scale AI's CliniCARE-Bench, built on 750 real patient cases across 25 clinical scenarios, found underlying models in agentic clinical systems reached error rates of 34.7%. When researchers required systems to demonstrate correct reasoning—not just correct answers—scores dropped by as much as 14.8 percentage points. The agents were bypassing longitudinal records, skipping conflicting evidence, and presenting conclusions that appeared earned but weren't, according to Francis deSouza, who published the findings this week.

In a benchmark environment, that's a research finding. In a live clinical trial, it's an undetected protocol deviation.

Why it matters

Clinical trials sponsors, contract research organizations, and health systems are deploying agentic AI for protocol eligibility screening, safety signal detection, and endpoint adjudication—decisions that directly affect patient outcomes. Yet 42% of health systems deploying clinical AI have no governance structure in place, according to a February survey of 120 institutions cited by cardiothoracic surgeon Mo Johnson. Most organizations treat administrative AI (scheduling, billing) and clinical AI (diagnostics, treatment planning) under identical oversight frameworks, despite vastly different failure costs.

The silent failure problem

Cassandra Chuljian identified the core risk: "The scariest AI agents aren't necessarily the ones that fail. They're the ones that seem to work." An agent running on stale data or flawed logic won't trigger alerts or deviation reports. It produces clean records that satisfy auditors until a patient outcome forces retrospective investigation—revealing process failures buried beneath correct-looking answers.

Current FDA guidance on AI-enabled medical devices, including the agency's 2021 action plan for machine learning-based software, has not operationally resolved this failure mode for agentic systems operating inside trial workflows.

Process versus performance

Research published this week by Sachin Bajpai in the International Journal of Computer argues the architecture conversation must happen before deployment. His framework, validated against sepsis-deterioration scenarios, separates thinking from acting layers and builds audit trails into decision architecture from the start—not as post-deployment compliance artifacts.

A randomized trial of LLM-based decision support in Kenyan primary care clinics, published in Nature Medicine and cited by UCSF's Sumant Ranji, demonstrated this disconnect: AI improved documentation of diagnoses and treatment plans but did not affect treatment failure rates. The process improved; outcomes did not.

Accountability architecture

Rubén Lozano Aguilera at Ai2 framed the core issue: "AI is not a moral agent; it can be reliable or unreliable, but it cannot be trustworthy." When Ai2's Asta AutoDiscovery flagged unexpected immune activity in invasive lobular carcinoma, scientists confirmed the signal in independent datasets and tumor tissue before committing to trials. The AI located the signal; humans defended the inference.

Ingrid O'Dwyer at Sanofi noted approximately 117 AI-discovered drugs have entered clinical trials, with discovery-to-Phase 1 timelines compressed from 4–4.5 years to 1.5–2 years. Speed is real. Clinical superiority has not been demonstrated.

The next regulatory action on AI-enabled clinical tools will cite a patient outcome, a missing audit trail, and a sponsor unable to explain how an agent reached its conclusion. The accountability infrastructure—who owns the failure, what audit trails must contain, how silent errors get detected—has not been designed for current deployment volumes.

These findings were first reported by Clinical Trial Vanguard, drawing on research from Scale AI, multiple health system surveys, and peer-reviewed clinical studies.