AI Tool Improved Clinical Notes in Kenya Trial, But Patient Outcomes Unclear

A clinical officer in Nairobi was about to send a feverish infant home when an AI system flagged an elevated heart rate and prompted her to examine the child's heart more carefully. The stethoscope revealed a whoosh indicating a possible congenital heart defect—a diagnosis that led to specialist referral, medication, and potential surgery.

That case illustrates both the promise and the limitations of a new AI clinical assistant tested across 16 Kenyan primary care clinics. The randomized trial, published in Nature Medicine, involved nearly 10,000 patient encounters and represents one of the first real-world evaluations of large language models supporting frontline healthcare workers.

How the System Works

The tool, called AI Consult, runs OpenAI's GPT-4o in the background while clinical officers type patient notes. It analyzes the documentation in real time and provides traffic-light-style alerts: green signals no issues, yellow indicates minor gaps or suggestions, and red flags critical concerns requiring immediate attention.

Vyonne Njeri, a registered clinical officer at Penda Health who participated in the trial, described the system as offering a "second pair of eyes." In settings where clinicians may see five or six patients per hour across a wide range of conditions—often without specialist backup—the prompts serve as reminders about protocols and guidelines.

The intervention cost just 4 cents per patient.

Better Notes, Uncertain Clinical Impact

An independent panel of six Kenyan family physicians reviewed the clinical notes and determined that providers using AI Consult produced higher-quality diagnoses and treatment plans compared to the control group.

However, the study could not demonstrate statistically significant improvements in patient outcomes. While treatment failures—including death or unresolved symptoms—decreased by 23 percent, the baseline rate of such events was too low to reach statistical significance.

Dr. Bilal Mateen, chief AI officer at PATH and a co-author of the study, explained that detecting a meaningful difference in treatment failures would require a trial of approximately 139,000 participants. "Treatment failures are just too rare in primary care," he noted.

Why It Matters

This trial moves beyond simulated testing to evaluate AI's role in actual clinical workflows, particularly in resource-constrained settings. The findings suggest AI can enhance clinical decision-making quality at minimal cost, even when direct evidence of improved patient outcomes remains elusive. For healthcare systems facing workforce shortages and limited specialist access, tools that strengthen primary care quality represent a meaningful step forward—though researchers acknowledge that proving mortality or morbidity benefits will require much larger studies.

Practical Use and Future Directions

Njeri reported finding the AI recommendations helpful about half the time, with the remainder being less useful but rarely incorrect. The expert panel rated most recommendations as safe and appropriate.

Dr. Jonathan Chen, an associate professor at Stanford University who was not involved with the research, praised the study for testing AI in real clinical settings rather than simulations. He suggested the technology's greatest impact may come from improving access to care—potentially by drafting clinical notes for provider review or even interacting directly with patients.

Mateen indicated he is working to implement the tool in healthcare systems globally. However, Dr. Nicholas Okumu, an orthopedic surgeon at Kenyatta National Hospital, cautioned that oversight must remain active as "even AI that's approved can still cause harm."

The trial was funded by the Gates Foundation. These details were first reported by NPR.