AI in Federal Rulemaking Challenges Core Judicial Review

The Arkansas algorithm disaster

When Arkansas automated Medicaid assessments in 2016, the results were catastrophic. An algorithm reduced home care for an amputee because the patient had "no foot problems." Nearly half of Arkansas Medicaid recipients saw their benefits cut by a system that couldn't distinguish between relevant medical needs and literal interpretation of data fields.

That failure foreshadowed a broader legal challenge now confronting federal courts. As AI use in federal agencies doubled from 2023 to 2024—with generative AI deployments increasing nine-fold, according to the Government Accountability Office—judges face a fundamental question: How can they review agency decisions when the reasoning behind those decisions is locked inside opaque algorithms?

Why it matters

The Administrative Procedure Act requires courts to strike down "arbitrary and capricious" agency actions, but that standard assumes human decision-makers whose reasoning can be examined. AI's black box nature threatens this core accountability mechanism just as the Trump Administration signals plans to introduce AI into the rulemaking process itself. Without adequate judicial review, agencies could make consequential regulatory decisions based on factors Congress never intended them to consider—and courts would have no way to detect or remedy the problem.

The black box meets administrative law

The Supreme Court has established two key principles for evaluating agency decisions: agencies cannot rely on "improper factors" that Congress didn't intend them to consider, and they cannot "entirely fail to consider an important aspect" of a regulatory problem. Both principles become nearly impossible to enforce when AI is involved.

An agency using AI cannot know exactly what factors the algorithm considered or how it weighted them. When Amazon built an AI hiring tool, the system developed gender bias by training on predominantly male resumes—penalizing applications that simply included the word "women's." Amazon's team ultimately scrapped the project because they couldn't guarantee the model wouldn't exhibit some form of bias.

If a federal agency used such a model to evaluate drug applications or environmental permits, it would be relying on improper factors without knowing it. The Food and Drug Administration, for instance, cannot determine which variables an AI model weighs when assessing a new pharmaceutical.

The third-party data problem

The challenge deepens when agencies procure commercial AI systems. The Government Accountability Office found that over half of surveyed agencies primarily buy AI products rather than building them internally. Commercial licensing terms typically shield training data and model architecture from government scrutiny.

Modern large language models train on massive internet datasets, including social media posts and user-generated content. ChatGPT, for example, incorporates Reddit data. These sources contain racist remarks, gender stereotypes, and other content that would be clearly impermissible if a human decision-maker explicitly considered them. Yet this material becomes statistically embedded in models that agencies then use for regulatory decisions.

This isn't merely a risk that AI might infer protected characteristics from proxy variables—it's the reality that models trained on third-party data are actually ingesting concrete information about race, gender, and other factors Congress prohibited agencies from considering. The result could violate both the Administrative Procedure Act and the Constitution's Equal Protection Clause.

Why existing workarounds fall short

Courts have long dealt with a similar black box problem in human decision-makers. An agency might construct a legally permissible rationale for a decision actually reached through improper considerations. Courts address this by scrutinizing agency procedures as a proxy for reasoned decision-making.

But several factors make that workaround inadequate for AI. Human imperfection in agency decision-making is unavoidable; AI use is not. Federal employees take oaths of office, face civil service accountability, and operate under whistleblower protections and Freedom of Information Act transparency requirements. None of these safeguards apply to algorithms.

These details were first reported by The Regulatory Review. Legal scholars are now looking to existing frameworks around government outsourcing to third parties for potential solutions that could restore the accountability and evidence-based decision-making that administrative law seeks to ensure.