AI Medical Scribes Cost Doctors Time Instead of Saving It

AI tools designed to lighten physicians' administrative burden are creating new work instead

Artificial intelligence was supposed to free doctors from tedious documentation, letting them focus on patient care. A large-scale study from Dartmouth College shows the opposite is happening: physicians are spending significant time correcting AI-generated mistakes and filling in clinical gaps the technology misses.

Researchers analyzed 146,000 conversations between 10,105 patients and their primary care physicians at Dartmouth Health, a large rural health system. The study, presented at the 2026 Annual Meeting of the Association of Computational Linguistics, examined how AI drafts responses in online patient portals and compared those suggestions to what physicians actually wrote.

The team tested responses from six major AI systems: Claude, Gemini, ChatGPT, Llama, Aloe, and Qwen. They developed a specialized evaluation tool to measure alignment between AI outputs and real physician communications.

The clinical judgment gap

"We find that AI can sound like a doctor but not think like one," said Sarah Preum, PhD, the study's corresponding author.

The research identified three recurring problems: AI responses were frequently too long, included irrelevant or inaccurate medical information, and failed to ask appropriate follow-up questions that physicians would naturally pursue.

One case illustrated the stakes clearly. When a 32-year-old woman taking acid reflux medication reported constant nausea, the AI suggested dietary adjustments. The physician overrode that recommendation and asked whether the patient might be pregnant—a clinically essential question the AI missed entirely.

Why it matters

This research challenges the widespread assumption that generative AI will automatically reduce physician burnout and administrative overhead. Health systems are investing heavily in AI scribing tools, but if those systems require extensive human correction, they may worsen the time crunch rather than solve it. The findings suggest that current AI models lack the clinical reasoning needed to safely handle nuanced patient communications without close physician oversight.

Training AI to think clinically

The Dartmouth team's evaluation framework could point toward solutions. By systematically comparing AI outputs to expert physician responses across thousands of real cases, researchers can identify specific reasoning gaps and train models to better match clinical judgment.

The study represents the first large-scale analysis of AI performance in actual patient portal communications, moving beyond theoretical capabilities to measure real-world impact on physician workflows.

The research was first reported by Inc. and published in the proceedings of the Association of Computational Linguistics conference.