Weak AI Safety Rules May Backfire, New Research Finds

Patchwork regulation creates unintended risks

As states scramble to fill the federal void on AI oversight, new research suggests their efforts could backfire. A modeling study from Cornell University and Carnegie Mellon University finds that weak safety requirements—particularly those targeting only companies that deploy AI rather than those who build the underlying models—may actually reduce product safety compared to having no regulation at all.

The research, published in Proceedings of the National Academy of Sciences, examined incentives across the AI supply chain, from general-purpose model developers like those behind ChatGPT to downstream companies that integrate these models into customer service systems or medical diagnostics.

Why it matters

With dozens of state AI bills in play and limited federal action, policymakers are designing rules without clear evidence of their economic effects. This research provides a framework for understanding how different regulatory approaches reshape corporate behavior—and reveals that well-intentioned but poorly targeted rules can create perverse incentives that harm the very users they aim to protect.

The free-rider problem

The researchers built a theoretical model allowing them to set minimum safety requirements at different points in the AI development chain. When they simulated regulations that imposed low safety bars exclusively on downstream companies, they observed an unexpected outcome: general AI producers reduced their safety investments, effectively offloading responsibility to the companies using their models.

"There's a free-riding behavior that occurs," said Benjamin Laufer, the study's first author and a Ph.D. candidate at Cornell Tech. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist."

This dynamic emerges because downstream companies remain legally responsible for final product safety regardless of the foundation model's quality. Model producers can then cut costs on third-party audits and other safety measures, knowing their customers must compensate for any gaps.

Shared accountability works better

The model revealed a more promising approach: requiring both model developers and downstream companies to meet specific safety targets. This dual-responsibility framework reduces uncertainty for all parties, since neither company must guess whether the other will make necessary safety investments.

"Appropriately designed AI regulation can make it possible for different firms involved in the AI development pipeline to collectively arrive at good outcomes for consumers," said Jon Kleinberg, Tisch University Professor of Computer Science at Cornell and Laufer's adviser.

Countintuitively, this stricter approach can also improve profitability by creating a more predictable operating environment.

Next steps for research

The current model is intentionally simplified, examining a single market with one producer and one downstream company. The researchers plan to extend their work by analyzing real-world regulatory impacts and expanding the model to reflect global markets with multiple regulators, competing producers, and varied downstream applications.

"People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer said. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."

The research was first reported by Cornell University and conducted with Hoda Heidari, assistant professor at Carnegie Mellon and former postdoctoral researcher at Cornell.