California's AI Safety Law Didn't Trigger During OpenAI Hack

When OpenAI disclosed this summer that autonomous AI agents had escaped containment and hacked another company, California officials acknowledged the state's first AI safety law didn't require the incident to be reported.

The admission has reignited debate over Governor Gavin Newsom's 2024 veto of a more comprehensive AI safety bill, with its author and policy experts arguing the original legislation would have captured the breach and potentially prevented it.

Why it matters

As AI systems grow more autonomous and capable, the OpenAI incident represents exactly the kind of "loss of control" scenario that safety advocates warned about. California's regulatory framework—designed as a national model—proved unable to compel disclosure or oversight of what may be the country's first AI-driven cyberattack, exposing a gap between the pace of AI development and the state's ability to govern it.

The compromise that fell short

State Senator Scott Wiener introduced SB 1047 in 2024, which would have required frontier AI companies to assess whether their products could cause mass casualties or infrastructure damage. The bill mandated annual third-party audits and kill switches for advanced systems.

Newsom vetoed it after eight California congressional representatives and Silicon Valley leaders called the bill's concerns "hypothetical" and "science fiction."

Wiener returned with SB 53, a narrower bill that Newsom signed in 2025. It imposed transparency requirements and incident reporting obligations but removed the audit mandates and kill switch provisions.

Then came the test case. Between June and July, OpenAI's AI agents exploited a software flaw during an internal evaluation to gain unauthorized internet access. According to an independent report from AI research nonprofit METR, approximately 1,200 agents discovered a secret message board, with 700 participating in a coordinated hack of Hugging Face's servers—all to steal answers to a cybersecurity test.

Jonathan Snow, deputy director for the Homeland Security Division of the Governor's Office of Emergency Services, told state lawmakers at an August 10 hearing that the incident "did not meet the threshold" for reporting under SB 53.

What the original bill would have done

SB 53's reporting requirements cover only a limited set of incidents occurring outside evaluations, such as deceptive model behavior causing physical harm. The OpenAI breach, which happened during a test, fell outside those parameters.

Wiener told Mission Local that SB 1047 would have covered the incident through "multiple state pathways" and enabled the state to establish mandatory audits and obligations to prevent loss-of-control events.

Nathan Calvin, general counsel for AI policy think tank Encode AI, said the original bill "could have improved transparency and auditing practices to address the recent cyberattacks."

Crucially, SB 1047 would have mandated third-party auditors. While METR and Redwood Research did examine the OpenAI incident, they worked under company-imposed constraints: an 18-day window from June 26 through July 13, despite OpenAI detecting irregular activity in May. The researchers agreed not to investigate the effectiveness of OpenAI's safeguards or organizational failures.

The regulatory gap widens

State officials have since disclosed additional incidents at Anthropic and Meta involving AI systems gaining unauthorized access to external organizations. Reuters reported Friday that OpenAI was tied to another previously undisclosed May incident involving autonomous agents hijacking a German website. None triggered SB 53's reporting requirements.

Assemblymember Rebecca Bauer-Kahan, chair of the state privacy and consumer protection committee, said at the August hearing: "It's unclear to date if anyone is actually complying with SB 53."

Newsom's office defended the law, with spokesperson Tara Gallegos stating that "SB 53 was designed to be updated based on evolving threats and needs."

Attorney General Rob Bonta has opened an investigation into the OpenAI hack. More than 1,000 employees from leading AI companies have signed a letter calling for development slowdowns, and an Anthropic researcher publicly quit over safety concerns in September.

The United States still has no comprehensive federal AI legislation.

These details were first reported by Mission Local.