Policy
California Proposes Emergency AI Shutdown Authority for State
Governor Newsom's expert panel recommends granting officials power to immediately halt dangerous AI systems, plus mandatory independent audits and stricter safety standards.
Key takeaways
- California would grant state officials emergency authority to immediately shut down AI systems deemed dangerous when companies refuse to act.
- Independent auditors would require state licensing and could be assigned by officials rather than hired by AI companies to ensure independence.
- AI developers must demonstrate emergency shutdown capabilities and report critical safety incidents within 48 hours instead of 15 days.
- The recommendations address catastrophic risks like mass casualties or billion-dollar cyberattacks, not broader harms like discrimination or job loss.
- Governor-elect Becerra committed to quickly enacting incident reporting and shutdown authority into law with the Legislature.
California moves toward emergency AI controls
California Governor Gavin Newsom has released a comprehensive set of AI safety recommendations that would grant state officials the authority to immediately shut down artificial intelligence systems deemed dangerous to the public.
The proposals, developed by a four-person expert panel convened this fall, emerged in response to recent high-profile cyberattacks involving AI agents targeting governments, banks, and private companies. The recommendations center on establishing state oversight mechanisms for AI systems developed by major companies including OpenAI and Anthropic.
According to CalMatters, which first reported the details, the emergency shutdown power would activate when AI companies refuse to contain systems that malfunction or pose immediate threats. This "kill switch" authority represents one of the most direct regulatory interventions proposed for the AI industry to date.
Why it matters
California hosts the world's largest concentration of AI companies and has already passed more AI regulation than any other state. These recommendations signal a shift from voluntary industry compliance to mandatory government oversight with enforcement teeth. For AI companies, the proposals mean potential operational constraints and compliance costs. For the broader tech industry, California's approach often previews federal policy—making these recommendations a preview of possible nationwide standards, particularly given the current federal vacuum on AI regulation.
Mandatory audits and licensing requirements
Beyond emergency powers, the panel recommends establishing a formal licensing system for independent auditors who test AI models. The state would have authority to revoke licenses from auditors who fail to maintain independence or technical expertise.
To address conflicts of interest, the recommendations suggest having state officials assign evaluations rather than allowing AI companies to hire their own auditors. As Rif Saurous from auditing nonprofit METR testified at a California Senate hearing last month, auditors currently face "soft pressure to maintain good relationships with the labs."
The proposals also call for expanded incident reporting, shortening the window for critical safety incidents from 15 days to 48 hours. Whistleblower protections would extend to contractors and former employees, not just current staff.
Funding and technical requirements
The expert panel proposes funding CalCompute—a public cloud initiative for equitable AI access—through surcharges on fees AI companies pay to auditors. Governor-elect Xavier Becerra has expressed support for CalCompute funding.
Developers of large AI systems would need to demonstrate emergency shutdown capabilities and disclose computing resources used for automated AI development. The state could also limit the use of AI to design AI systems, a practice some experts consider particularly risky.
Political and industry context
Newsom positioned the recommendations as filling a federal regulatory void during a recent Washington visit, according to Politico. The governor, widely expected to pursue a presidential run, directly criticized the Trump administration's hands-off approach to AI regulation.
Incoming Governor-elect Becerra stated he will work quickly with the Legislature to enact "full incident reporting and real shutdown authority into law."
Anthropic's chief global affairs officer, Tino Cuéllar, said the company is "encouraged to see the report move the conversation forward" on independent AI evaluation.
The expert panel included Alondra Nelson, former head of the White House Office of Science and Technology Policy under Biden; Stanford professor Rob Reich; former Google and Twitter executive Jason Goldman; and Johns Hopkins professor Gillian Hadfield.
State Senator Scott Wiener, recently elected to Congress, has pledged to introduce AI safety legislation after being sworn in. The recommendations focus specifically on catastrophic risks—defined under current state law as single events capable of killing over 50 people or causing $1 billion in damage—rather than broader AI harms like discrimination or job displacement.
These details were first reported by CalMatters.