Healthcare AI Shifts From Experimentation to Workflow Governance

Healthcare organizations are entering a new phase of artificial intelligence adoption, moving from general-purpose experimentation to production systems designed specifically for clinical workflows and accountability, according to Manu Agrawal, chief architect leading AI and machine learning initiatives at Oracle Health.

The shift is marked by the emergence of healthcare-specific platforms such as ChatGPT for Healthcare and Claude for Healthcare. These tools signal that the industry is no longer satisfied with eloquent text generation alone — providers need systems that understand clinical terminology, coding standards, privacy requirements, and workflow context while fitting into existing operations.

Why it matters

As healthcare AI moves from pilot projects to production environments, the industry faces a critical question: how do organizations ensure these systems are safe, auditable, and appropriate for the clinical decisions they support? The answer will determine which platforms deliver lasting value and which create new risks. Organizations that focus solely on model performance without building governance infrastructure may find themselves unable to deploy AI at scale or defend its outputs when clinical decisions are questioned.

Beyond benchmarks and demonstrations

Agrawal, speaking in a personal capacity, argues that sophisticated models alone won't guarantee successful deployment. The foundation remains data quality — organizations need confidence in where information originated, how current it is, which coding systems apply, and whether the model accesses only appropriate data.

"A model's answer is only as trustworthy as the data, context and controls behind it," she said, as first reported by Healthcare IT News.

Production systems often combine deterministic rules, retrieval mechanisms, policy logic, clinical algorithms, and generative AI rather than relying on a single model. This hybrid approach reserves generation for suitable tasks while using predictable tools for compliance and workflow interpretation.

Different EHR implementations, documentation practices, payer requirements, and local policies can change the meaning of an otherwise straightforward request. The same question may require clinical context, administrative context, coding logic, and patient-specific constraints.

Risk-calibrated oversight

Agrawal recommends tailoring governance to the risk level of each workflow. Low-risk activities such as documentation support or context gathering may justify greater automation, while decisions affecting patient safety — diagnosis, autonomous triage, therapy selection, medication changes — require stronger safeguards.

"The critical capability is not simply keeping a human in the loop everywhere," she said. "It is knowing where human-in-the-loop review is required, why it is required and how that review is captured."

Near-term opportunities concentrate in operational and administrative work where AI can reduce manual effort: cohort building, prior authorization, patient communication, care management, and trial feasibility. The measurable value comes when technology connects scattered data to the right context and reduces the time clinicians and administrative teams spend gathering information.

What enterprise leaders should ask

For CIOs, CMIOs, and AI governance leaders evaluating platforms, Agrawal recommends looking beyond impressive demonstrations and benchmark scores. Key questions include whether systems can be governed at the workflow level, whether permissions align with clinical roles, and whether every response can be traced to sources.

Auditability should allow organizations to reconstruct transactions later: what information was available, what the AI produced, which evidence it used, who reviewed it, what was changed, and what action followed. Without that record, accountability becomes difficult when clinical or operational decisions are questioned.

Validation should be tailored to each workflow, with acceptance criteria set before deployment and tested against local data, populations, and workflows. A documentation assistant, prior authorization tool, and clinical evidence application should not share identical success metrics because each carries different risks.

Ultimately, Agrawal said organizations should determine whether a system can be "safe, useful, monitored, auditable and governed for the specific workflow where we are deploying it." That shift from model fluency to operational accountability will determine which healthcare AI platforms deliver lasting value.

These details were first reported by Bill Siwicki in Healthcare IT News.