Enterprise
AI Message Drafts Don't Save Doctors Time, UC San Diego Study Finds
Randomized evaluation shows physicians spend as long editing AI-generated patient replies as writing from scratch, though cognitive burden drops.
Key takeaways
- UC San Diego Health's randomized trial found AI-drafted patient message replies did not reduce physician time, though cognitive burden decreased
- Frequent low-effort edits for scheduling and referrals created the largest cumulative workload, while interpretation and dosage edits were most time-intensive individually
- Edit rates remained stable over 17 months with no evidence of automation bias or declining physician vigilance
- Health systems should use physician editing patterns to guide AI development rather than imposing rigid guardrails that produce errors
- Current AI drafting tools target workforce well-being rather than direct clinical outcomes or measurable time savings
AI-drafted patient messages fail to deliver time savings
Physicians using artificial intelligence to draft patient message responses spend just as much time on the task as those writing replies manually, according to a randomized evaluation conducted at UC San Diego Health.
The health system was among the first to integrate AI-drafted message replies directly into its electronic health record in 2023, positioning researchers to measure real-world impact on clinical workload. While the AI tools did not reduce time spent on messaging, physicians reported experiencing lower cognitive burden when using the drafts, according to Karandeep Singh, MD, MMSc, Chief Health AI Officer at UC San Diego Health.
Patient messages represent a growing share of physician work and correlate with burnout. The findings, first reported by the NIH Collaboratory's Rethinking Clinical Trials initiative, suggest health systems must look beyond simple time savings when evaluating AI clinical tools.
Where AI drafts fall short
Following the randomized trial, Singh's team conducted a quality improvement study analyzing physician edits to understand where AI-generated drafts require the most correction. The research tracked what physicians changed, the extent of modifications, and the time cost of each edit type.
The analysis revealed that frequent, low-effort edits—including scheduling coordination, lifestyle advice, and referral management—accumulated into the largest overall burden. Meanwhile, edits involving clinical interpretation, clarification of medical information, or medication dosages proved most time-intensive on an individual basis.
Over 17 months of monitoring, edit rates and intensity remained consistent, showing no evidence that physicians were becoming overly reliant on AI output without proper review—a phenomenon known as automation bias.
Implications for health system AI strategy
Singh emphasized that health systems can use physician editing patterns to both steer AI development and monitor output quality. The approach allows organizations to target the most costly edit types for improvement while automating or delegating the most frequent corrections.
The research team noted that health systems should design system prompts to align with actual physician behavior rather than imposing rigid AI guardrails that can produce unintended errors. Given that AI vendors update models frequently, evaluators must rely on contemporary controls and monitor behavioral changes over time to assess tool quality accurately.
Why it matters
Current AI drafting tools primarily address physician well-being rather than direct clinical outcomes or operational efficiency. For health systems investing in AI to reduce documentation burden, this research suggests the technology may improve the experience of clinical work without yet delivering measurable productivity gains. Organizations will need to automate downstream clinical tasks and delegate protocolized workflows to achieve meaningful improvements in both outcomes and efficiency. The editing analysis provides a practical framework for identifying where AI assistance creates value versus where it simply shifts work.
The findings were presented at a Grand Rounds session hosted by the NIH Collaboratory Rethinking Clinical Trials initiative in November 2024.