U.S. Health Agencies Doubled AI Use in Two Years Amid Safety Concerns

Federal health agencies dramatically accelerated their adoption of artificial intelligence between fiscal years 2024 and 2025, according to the U.S. Department of Health and Human Services AI Use Case Inventory. The FDA experienced a 148 percent increase in AI applications, the CDC saw 87 percent growth, CMS expanded use by 78 percent, and the NIH increased AI deployment by 51 percent.

The expansion comes as agencies pursue efficiency gains and cost savings. But the rapid deployment raises a fundamental governance question: Are there healthcare decisions that should never be fully automated, or that require mandatory human oversight even when AI is involved?

From simple algorithms to black-box systems

The technology health agencies deploy has evolved significantly. In 2020, states used transparent rule-based algorithms to allocate scarce monoclonal antibody supplies during the COVID-19 pandemic. These systems were deterministic and auditable—their logic could be examined and challenged in court.

By 2023, during the Medicaid unwinding, states relied on more complex eligibility determination systems. Programming errors and data flaws led to hundreds of thousands of improper disenrollments. A class action against Tennessee's TennCare alleged violations of the Medicaid Act and due process stemming from its computerized eligibility system. The National Health Law Program filed a separate FTC complaint alleging that Deloitte Consulting's defective eligibility software caused widespread improper terminations in Texas, though the software was used by 20 other states.

Since 2023, agencies have increasingly adopted machine-learning models and generative large language models. A U.S. Government Accountability Office report found that while overall AI use cases across 11 federal agencies doubled from 2023 to 2024, generative AI use cases increased nine-fold, with health agencies leading adoption.

Documented harms in healthcare AI

Real-world failures demonstrate the stakes. Beneficiaries filed a class action alleging that UnitedHealth Group used an AI program to deny medically necessary post-acute care in Medicare Advantage. In March 2026, a federal district court ordered defendants to disclose the AI's error rates, which plaintiffs alleged reached 90 percent—meaning nine of 10 appealed denials were ultimately reversed.

Studies have documented diagnostic errors, unsafe triage advice, privacy breaches, and questionable applications in public sector decision-making. As AI systems become more complex and opaque, errors become harder to detect and trace. In healthcare, even rare errors can have serious consequences for patients.

A proposed framework for AI governance

Existing federal frameworks require impact assessments for "high-impact" AI uses but assume deployment will proceed. These assessments focus on mitigating risks rather than determining whether AI should be used at all.

Diane Hoffmann, Director of the Law & Health Care Program at the University of Maryland School of Law, proposes a Five Factor Framework requiring agencies to examine: the interests at stake and potential harms; the nature and normative structure of the decision itself; the characteristics and limitations of the proposed technology; the role of human judgment and oversight; and the legal constraints governing the decision, including due process and civil rights protections.

If an agency concludes AI benefits outweigh risks for a high-impact decision, Hoffmann argues it should seek public input through notice and comment, town halls, or engagement with advocacy groups representing affected communities.

Why it matters

Without principled frameworks for determining when automation is appropriate, health agencies risk deploying AI systems that make errors at scale in decisions affecting patient safety, healthcare access, and civil rights. The current approach prioritizes efficiency without systematically weighing whether certain governmental functions require human judgment that AI cannot replicate. As generative AI adoption accelerates, the gap between technological capability and governance frameworks widens.

These details were first reported by the Petrie-Flom Center at Harvard Law School in an analysis by Diane Hoffmann.