California Now Reports Six High-Risk AI Systems After Zero Last Year

California state agencies are now reporting the use of six high-risk automated decision systems that affect residents' lives, a sharp reversal from last year when officials claimed they used none.

The systems currently in operation include tools that predict whether incarcerated individuals will reoffend, evaluate unemployment claims for fraud, remotely proctor exams for California State University students, and detect AI-generated content in student assignments. The disclosure came in a report released by the state's technology department, as first reported by CalMatters.

Why it matters

The dramatic shift from zero to six reported systems in one year raises questions about government transparency and oversight of AI tools that make consequential decisions about housing, employment, healthcare, criminal justice, and education. Several of these systems have been in use for years—including COMPAS recidivism scoring software used by corrections officials for at least a decade—yet went unreported in the initial inventory.

What changed in the reporting process

State officials attributed the increase to a more thorough evaluation process. According to the report, the technology department conducted direct meetings with agencies and questioned them more extensively about their systems, rather than relying solely on written responses.

A 2023 state law requires annual disclosure of "high-risk automated decision systems" defined as tools used to assist or replace human decisions that have legal or similarly significant effects on people's lives. The legislation was championed by civil rights and privacy advocates concerned about algorithmic bias, particularly against marginalized communities.

Last year's report drew scrutiny after CalMatters noted obvious omissions, including the corrections department's recidivism prediction software and an employment department fraud detection system that froze benefits for 600,000 Californians during the 2020 holiday season.

Notable gaps remain

The latest inventory still excludes several known AI deployments. Missing from the report are generative AI pilot projects backed by the governor's office, including Poppy—an AI assistant using Anthropic's Claude model that 67 state departments tested and which is scheduled for statewide rollout next month.

Also absent is California State University's contract with OpenAI to provide ChatGPT access. The 2023 law exempts certain entities from reporting requirements, including the judicial branch and University of California system, despite CalMatters reporting that a majority of the state's roughly 60 courts have adopted generative AI policies.

The report identified two additional high-risk systems not currently active: the Department of Cannabis Control is developing AI to analyze marijuana packaging for child appeal violations, while California State University discontinued using a language model for job application reviews.

State agencies initially flagged six other systems as high-risk but later determined they did not meet the threshold, including AI used by the Department of Finance for legislative bill analysis.

Details of California's AI inventory were first reported by CalMatters, which is compiling a comprehensive database of automated decision systems used by state and local agencies throughout California.