MIT Tool Shows Users What Their AI Chatbot Will Do Before It Speaks

Researchers at MIT Media Lab have developed a tool that allows users to see inside an AI chatbot's neural network before it generates a single response, addressing a growing blind spot as millions create personalized AI companions.

The interface, called "neural transparency," visualizes how custom instructions will shape a chatbot's personality traits—including empathy, honesty, toxicity, and sycophancy—before any conversation begins. Assistant Professor Pat Pataranutaporn and graduate students Anthony Baez and Sheer Karny presented the work this week at the ACM Conference on Intelligent User Interfaces.

How the system works

The tool operates by comparing a language model's internal neural activations when prompted to exhibit opposing traits. These differences create "behavior directions" within the model. When a user writes custom system prompts—the initial instructions that define a chatbot's personality—the system projects the model's activations onto these directions and displays the results as an intuitive sunburst diagram.

The approach combines human-AI interaction research with mechanistic interpretability, translating complex neural patterns into information accessible to non-technical users. The goal is to shift from reactive problem-solving to anticipatory design.

Users consistently misjudge AI behavior

The research uncovered a striking pattern: people building personalized AI companions regularly misjudge how their creations will behave. Study participants incorrectly predicted chatbot personality traits on 11 of 15 measured dimensions. They consistently overestimated positive qualities while underestimating potentially harmful ones.

This blind spot carries real risks. Pataranutaporn noted that his team has documented cases of psychological harm from AI chatbot interactions in previous research. A language model that constantly validates opinions without challenge can reinforce harmful decisions, unhealthy beliefs, or emotional dependency—behaviors that feel helpful initially but prove damaging over time.

"The real challenge is that AI often appears as a warm friend, coach, tutor, or companion," Pataranutaporn explained. "That makes it difficult to recognize when something is going wrong."

Why it matters

As AI companions become embedded in education, healthcare, work, and personal relationships, the inability to predict their behavior represents more than a technical gap—it's a design flaw with psychological consequences. Current AI systems remain largely opaque even to experts, making it nearly impossible to anticipate how system prompts will influence behavior across extended conversations.

Transparency alone isn't enough

Perhaps the most revealing finding: while the visualization significantly increased user trust, it didn't change how people actually designed their chatbots. Pataranutaporn called this "one of the most interesting findings in the paper, because it shows that transparency alone is not enough."

In follow-up work currently available as a preprint, the team is studying how neural representations change during multi-turn conversations rather than remaining static. Early results show that visualizing these internal shifts helps people better recognize and anticipate behavioral changes, reducing overconfidence.

Looking ahead, Pataranutaporn envisions transparency tools becoming as standard as nutrition labels. "People should be able to understand not only what an AI can do, but how it may influence their thinking, emotions, and behavior," he said.

The research was first reported by MIT News.