Health Systems Deploy AI Governance as Shadow Tools Proliferate

Health Systems Deploy AI Governance as Shadow Tools Proliferate

Artificial intelligence is moving rapidly from pilot programs into daily clinical operations at major U.S. health systems, but leaders warn that scaling these tools safely requires more than enthusiasm. It demands formal governance, clean data infrastructure, and the operational discipline to redesign workflows rather than simply automate broken processes.

At the Becker's Annual Meeting in Chicago, executives from Atlantic Health, Ochsner Health, Rush University System for Health, and Sanford Health described a healthcare AI landscape shifting from experimentation to enterprise strategy. A central challenge: the proliferation of "shadow AI"—unsanctioned tools that physicians and staff adopt on their own to solve immediate workflow problems.

Why it matters

As AI agents begin handling patient scheduling, clinical documentation, and predictive screening, health systems face a critical inflection point. The technology is advancing faster than most organizations can absorb it, creating gaps in cybersecurity, data quality, and accountability. Without structured governance, systems risk fragmenting their AI deployments, exposing patient data, and failing to capture measurable value from their investments.

Shadow AI reveals unmet workflow needs

Dr. Louis Jeansonne, chief medical informatics officer at Ochsner Health, reframed the shadow AI problem. Rather than treating unsanctioned tools as misconduct, he said health systems should view them as workflow intelligence—signals that clinicians need better solutions.

An early example emerged when staff began using personal AI bots in Zoom meetings to take notes. "It made us realize that you can't just tell people they're not allowed to use it," Jeansonne said. "It's making their lives easier."

Ochsner's response: survey staff about which AI tools they're using, then provide approved alternatives before issuing prohibitions. The approach treats shadow AI as a discovery mechanism for genuine operational needs.

Patient privacy remains the primary risk. While most clinicians know not to enter patient names into ChatGPT, Jeansonne noted that combining enough contextual details—dates, times, personal information—can make data identifiable when cross-referenced with other datasets.

Governance structures must precede scale

Sunil Dadlani, chief information and digital officer at Atlantic Health, emphasized that governance structures are non-negotiable. Without them, AI adoption devolves into chaos and fragmentation.

Atlantic Health is working toward "native AI"—embedding intelligence across clinical operations, revenue cycle, and patient engagement. The system has documented measurable returns: reduced physician burnout, decreased after-hours documentation time, improved clinical coding accuracy, and better patient throughput.

One agentic AI application for colonoscopy scheduling reduced no-shows and cancellations by 40% in 30 days. About 10% of patients actively interacted with the AI agent, which Dadlani said helped build trust.

But he cautioned that technology creates value where it acts autonomously—and that's also where the highest risk concentrates. "You have to be very cognizant and careful about what the use case is and what kind of different flavor of AI that you want to deploy," he said.

Dadlani advised starting with the problem, not the vendor pitch. Identify system-level challenges first, then determine the appropriate solution.

Fix data and workflows before deploying agents

Jeff Gautney, chief information officer at Rush University System for Health, said AI exposes bad data and nonstandard processes. Rush learned this when deploying an AI agent to surface clinic hours. The system appeared to hallucinate incorrect information—but the real problem was that clinic hours were updated via email chains, leaving the data two or three days out of date.

"What looked like an AI hallucination was bad data—and a bad process behind the data," Gautney said. Rush is advancing AI deployments while simultaneously fixing foundational issues. Waiting for perfect data would freeze progress for years.

Gautney warned that operations leaders must learn to manage a digital workforce. Unlike traditional IT systems that fail abruptly, AI systems degrade slowly. Monitoring for drift, bias, and performance degradation is essential—and it's an operational responsibility, not just an IT task.

He also flagged cybersecurity concerns, noting that vulnerabilities have been discovered in systems monitored for years. "Don't assume that all technology partners will act as responsibly as you would like," he said.

AI closes access gaps in rural markets

Brad Reimer, chief technology and digital officer at Sanford Health, highlighted AI's role in bridging access gaps. Sanford's machine learning models for chronic kidney disease screening have doubled screenings and tripled early diagnoses by identifying high-risk patients in the 90% of the population not flagged by traditional criteria.

The impact is particularly significant in rural areas where chronic conditions are more prevalent and physician shortages are acute. "It's going to bridge the gap in a really meaningful way for patients," Reimer said.

All four health systems are members of the AMA Health System Member Program. Details on their AI governance approaches and deployment outcomes were first reported by the American Medical Association.