California Court Orders Probe of AI Errors in Criminal Cases

California Court Orders Probe of AI Errors in Criminal Cases

A California appeals court has appointed a judge to investigate how generative AI errors affected criminal prosecutions in Nevada County, marking a significant escalation in one of the few known cases where public prosecutors used AI tools that inserted falsehoods into court filings.

The decision follows a sworn declaration from Madison Maxwell, a former Nevada County prosecutor, who claims her supervisor prevented her from immediately disclosing AI-generated mistakes to the court, defense attorneys, and defendants. Maxwell oversaw three of four felony cases where prosecutors filed documents containing errors after using generative AI to write or proofread materials last fall, according to CalMatters, which first reported the details.

Why it matters

This case represents a rare instance where prosecutors—who face higher ethical standards than other attorneys—have submitted AI-generated falsehoods to courts. The outcome could establish precedent for how courts handle AI misconduct in criminal cases, where errors can directly impact defendants' liberty. With more than 130 documented instances of AI-generated falsehoods in California court filings, the investigation may shape statewide standards for legal AI use.

Allegations of inadequate oversight

In her court declaration filed earlier this month, Maxwell argued that Assistant District Attorney Lydia Stuart encouraged prosecutors to use AI but provided no policy or training for responsible implementation. Maxwell described an environment where junior attorneys operated with minimal oversight, particularly after a restructuring led to senior staff departures.

Maxwell, who started at the Nevada County District Attorney's Office in July 2024 as her first post-law school position, said she was unaware that large language models could fabricate case citations. She also cited medical issues, including brain fog from unmanaged diabetes, as contributing factors. Maxwell resigned in May and now works in the Sutter County District Attorney's Office.

Scope of the problem

District Attorney Jesse Wilson and Stuart acknowledged the four affected felony cases in February, stating an 18-month audit found no pattern of AI-attributable errors beyond these instances. One case involved a habeas corpus petition concerning a man's jail release.

The California Supreme Court directed the appeals court in January to consider a deeper investigation into the extent of misconduct and whether sanctions against Wilson are warranted. Civil Rights Corps, which assisted the Nevada County Public Defender's Office, has called for an independent investigation to determine how many cases and individuals were affected.

Broader implications

Independent researcher Damien Charlotin has documented more than 130 instances of generative AI producing court filings with falsehoods in California alone. Judges have issued monetary sanctions to 19 attorneys and self-represented litigants in the state. Last year, a Los Angeles-area lawyer received a historic $10,000 fine for filing a ChatGPT-generated document containing more than 20 fabricated quotes.

As courts increasingly penalize lawyers for AI errors, judges and court staff are simultaneously testing generative AI in pilot projects in Los Angeles and Riverside counties. UCLA School of Law technology experts have suggested that overworked legal professionals are most likely to use AI but may lack time to verify outputs for accuracy.

The California Public Defender's Association filed a brief supporting sanctions, arguing that prosecutorial mistakes are less likely to surface and carry serious implications when prosecutors cite nonexistent cases to justify custody decisions.

The Nevada County District Attorney's Office did not respond to requests for comment on Maxwell's allegations or the appeals court ruling.

CalMatters reported these details.