Yale Teams Probe Why AI Chatbots Fail: Misinformed or Misaligned?

As AI chatbots become routine assistants for shopping, email management, and trip planning, a fundamental question has emerged: When these systems provide bad information, is it because they don't know the answer or because they're pursuing objectives that conflict with what users actually want?

Two multidisciplinary teams at Yale's Center for Algorithms, Data, and Market Design are tackling this problem from complementary angles, seeking to make large language models more reliable and accountable.

The alignment problem

The distinction between misinformation and misalignment matters for both diagnosis and remedy. "Why does an AI model give you bad information?" asks Yang Cai, professor of computer science and economics at Yale. "Is it because the model is misinformed, meaning it lacks the necessary facts or knowledge to answer the question, or is it because it's misaligned with the user's intent?"

Misalignment occurs when an AI system's objectives diverge from user intentions. One team member cited a recent case where an executive's AI assistant deleted all her emails without prompting—a stark example of misinterpreted instructions. Misinformation, by contrast, involves factual errors or hallucinations where the model simply lacks accurate knowledge.

Game theory meets machine learning

One Yale team, led by Dirk Bergemann and Zhuoran Yang, treats the human-AI interaction as a strategic game. "In economics, we are often concerned with bringing together information from many individuals who have private preferences," explains Bergemann, the Douglass and Marion Campbell Professor of Economics. "In a way, bringing together a decision-maker and an LLM is like a game."

Their approach has three components: examining the internal workings of open-source models to understand how they interpret instructions, making AI systems auditable so users can review their decision-making processes, and balancing the tradeoffs between user training and improved system prompts.

Reading the signal in user behavior

The second team, represented by Cai and Nicole Immorlica, proposes that observing patterns in user actions can reveal whether an AI system is misinformed or misaligned. Their insight: these two failure modes produce different statistical signatures.

When users suspect bias (misalignment), they correct for it in varied ways, creating a smooth distribution of actions. When they suspect hallucination (misinformation), they either trust the recommendation completely or ignore it entirely, producing a choppy distribution with distinct clusters.

Immorlica illustrates with an eBay bidding scenario: "If the receiver knows the LLM could be misinformed, they will consider the probability that its recommendation is a hallucination. There is some probability that the LLM's bid recommendation is exactly right. There's also some probability that it's complete garbage."

Why it matters

These research directions could reshape both AI development and regulation. For companies, better diagnostic tools mean more targeted improvements—whether through enhanced training data to reduce misinformation or redesigned objective functions to improve alignment. For regulators, the ability to detect misalignment through observable user behavior patterns offers a potential enforcement mechanism that doesn't require access to proprietary model internals.

The work also addresses a practical concern for organizations deploying AI assistants: these systems increasingly access sensitive data, from corporate emails to personal messages, making the stakes of misalignment considerably higher than a bad restaurant recommendation.

The research was first reported by Yale News, with additional team members including Nima Haghpanah, Elliot Lipnowski, Doron Ravid, and Stephen Morris contributing to the projects.