FDA Proposes Doctor-Style Competency Tests for Medical AI

The Food and Drug Administration has released a discussion paper proposing a fundamental shift in how it evaluates generative AI-enabled medical devices, drawing inspiration from the way human physicians are credentialed and assessed.

The agency is exploring a "competency-based approach" that would establish performance benchmarks and require validation in clinical settings before AI tools can be used in patient care. This marks a departure from traditional medical device regulation, which the FDA acknowledges hasn't kept pace with the explosion of AI tools now directly involved in clinical decision-making.

Why it matters

The regulatory gap for AI medical devices has real consequences: hospitals and clinics are deploying tools that range from simple information aids to systems that make autonomous care decisions, yet there's no consistent framework for ensuring they work safely as they evolve. The FDA's proposal could establish a global template for how regulators handle AI that learns and changes over time.

Risk-based evaluation framework

The discussion paper, first shared with Axios, outlines a method for assessing AI device risk by measuring the type of clinical activity performed against the potential severity of incorrect outputs. This risk stratification would determine the level of scrutiny required before market approval.

Michelle Tarver, director of the FDA Center for Devices and Radiological Health, emphasized the agency's intent to create "a transparent process to inform the development of an approach that safeguards patients and consumers, advances innovation and serves as a potential model for regulators around the world."

The competency model

Under the proposed competency-based framework, generative AI devices would need to demonstrate performance against established benchmarks, similar to how medical residents must pass board examinations. The FDA suggests comparing AI performance to either a panel of qualified clinicians representing the standard of care or to a median practicing clinician.

This approach acknowledges a fundamental challenge: generative AI systems produce variable outputs and evolve over time, making them substantially different from traditional medical devices with fixed, predictable behavior.

Postmarket surveillance questions

The paper raises a significant policy question about whether the FDA should accept "greater premarket uncertainty" about an AI device's benefit-risk profile in exchange for more rigorous postmarket monitoring. This would represent a shift toward ongoing surveillance rather than front-loaded approval processes.

Acting FDA Commissioner Kyle Diamantas framed the effort as essential to U.S. leadership: "Artificial intelligence is transforming medicine, and the United States must lead in shaping how this technology is developed and used safely and responsibly."

The agency emphasized that the discussion paper is not formal guidance for device manufacturers and that it's soliciting stakeholder feedback as part of a preliminary effort. The FDA stated its ultimate goal is enabling "a nimble regulatory approach that employs least burdensome principles and allows patients timely access to safe and effective medical devices."

The details were first reported by Axios.