Radiology AI Detects 39% More Brain Aneurysms in Hospitals Than Radiologists Alone

AI performance depends heavily on clinical context

Artificial intelligence algorithms designed to detect brain aneurysms deliver substantially different results depending on where they're deployed, according to new research from the Neiman Health Policy Institute.

A prospective study of nearly 4,000 consecutive CT angiography scans at Northwell Health found that a commercial deep learning algorithm identified 55 true-positive intracranial aneurysms that radiologists missed—a 39% relative increase in detection compared to physician-only performance. However, the algorithm's utility varied sharply across clinical environments, with emergency and inpatient settings showing clear benefits while outpatient applications generated more false alarms than correct findings.

The research, published in the Journal of the American College of Radiology, examined scans collected in late 2023 at the New Hyde Park, New York-based hospital system. Researchers used an FDA-cleared algorithm from Aidoc, processing images in "shadow mode" so the AI operated in parallel without influencing radiologist decisions. About 5% of the scans showed intracranial aneurysms, which affect 3% to 4% of the general population and can prove fatal if they rupture before detection.

Why it matters

This study provides rare prospective evidence that AI deployment decisions should be guided by clinical context, not just validation metrics. Healthcare organizations investing in diagnostic AI need performance data from their actual care environments—not just vendor benchmarks—to understand where algorithms add value and where they may create workflow friction through excessive false positives. The finding that the same algorithm performs differently across inpatient, emergency, and outpatient settings has direct implications for implementation strategy and resource allocation.

Performance breakdown by setting

In inpatient environments, the algorithm identified 18 additional aneurysms while generating only seven false-positive alerts. Emergency department performance was similarly favorable. However, in outpatient settings, AI contributed just four additional detections while producing more false positives than true findings.

Matthew Barish, vice chair of radiology informatics at Northwell Health and study co-author, attributed the variation to case complexity. Higher-acuity settings involve more clinically complex examinations, creating opportunities for AI to serve as a complementary detection tool alongside human interpretation.

Overall, physicians and the algorithm agreed in over 96% of cases. The AI demonstrated higher sensitivity than radiologists alone—85% versus 72%—meaning it found more aneurysms that were actually present. Human readers showed higher precision at 93% versus the algorithm's 78%, meaning they were more likely to be correct when flagging a positive case. Both AI and radiologists performed similarly well at ruling out aneurysms when none were present.

Radiologists identified 30 true-positive aneurysms that AI missed. Independent expert neuroradiologists reviewed all discrepancies to establish ground truth, and 46 of the AI-only findings proved to be false positives.

Implementation implications

Elizabeth Rula, executive director of the Neiman Health Policy Institute, emphasized that healthcare organizations should evaluate AI based on how it improves physician performance and patient care in real-world use, not solely on results from original testing environments. The research underscores the importance of ongoing monitoring and evaluation after implementation.

The findings were first reported by Radiology Business and published in the Journal of the American College of Radiology.