Weak AI Safety Rules May Harm Consumers More Than No Rules

Regulators attempting to govern artificial intelligence may inadvertently make AI products less safe if they set standards too low or target the wrong parts of the supply chain, according to new modeling research from Cornell University and Carnegie Mellon University.

The study, published in Proceedings of the National Academy of Sciences, examined how different regulatory approaches affect incentives for companies building general-purpose AI models—like those powering chatbots—and downstream firms that deploy those models in customer service systems or medical diagnostics.

Why it matters

As states rush to fill the federal regulatory vacuum with their own AI laws, policymakers lack clear guidance on which approaches actually improve safety versus those that merely create compliance theater. This research suggests that well-intentioned but poorly designed rules could make consumers worse off by allowing upstream AI providers to shift safety responsibilities downstream while reducing their own investments in protective measures.

The backfire effect

The researchers discovered a counterintuitive outcome: when regulations set a low safety bar and target only downstream companies using AI models, the resulting products were predicted to be less safe than in completely unregulated scenarios.

"There's a free-riding behavior that occurs," said Benjamin Laufer, who completed his doctorate at Cornell Tech and led the study. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist."

This dynamic emerges because general AI producers can reduce investments in safety measures like third-party audits, knowing downstream developers remain legally responsible for the final product's safety. The model defined safety broadly to include any risk of user harm, such as toxic chatbot responses.

A regulatory sweet spot

The research identified more promising approaches. When regulations establish specific safety targets for both general AI producers and downstream companies, products become safer while potentially increasing profits for both parties.

This dual-target approach reduces risk by creating predictable expectations. Companies no longer need to guess whether their partners will make necessary safety investments—the regulation ensures both parties meet defined standards.

"Appropriately designed AI regulation can make it possible for different firms involved in the AI development pipeline to collectively arrive at good outcomes for consumers," said Jon Kleinberg, Tisch University Professor of Computer Science at Cornell, who advised the research alongside Carnegie Mellon assistant professor Hoda Heidari.

Next steps

The current model represents a simplified view of AI regulation. The researchers plan to examine real-world regulatory impacts and expand their framework to account for multiple regulators setting different standards across jurisdictions, as well as competition among AI providers and downstream companies.

"People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer noted. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."

The findings were first reported by Cornell University and published July 20 in Proceedings of the National Academy of Sciences.