AI Confidence and Accuracy Stem from Separate Internal Features

Separate pathways for confidence and correctness

Large language models frequently deliver wrong answers with unwavering confidence or hedge on responses they actually get right. New research from UC Riverside computer scientists reveals why: confidence and correctness emerge from distinct internal features within AI models, not from a single unified mechanism as commonly assumed.

The finding challenges a core assumption in AI development—that a model's expressed certainty reliably indicates accuracy. More importantly, it points toward practical methods for building systems that know when to be confident and when to hold back.

Why it matters

As organizations deploy large language models for decision support and task automation, determining when AI outputs can be trusted becomes critical. This research provides a roadmap for making models more reliable without the prohibitive cost of complete retraining, offering developers specific internal features they can adjust to align confidence with correctness.

Examining what happens inside the model

Het Patel, a doctoral student in computer science at UC Riverside and lead author of the study, explained that his team went beyond documenting the confidence-accuracy mismatch. They identified which internal features activate when models exhibit particular behaviors.

Working with Meta's Llama-3.1-8B and Google's Gemma-2-9B—open-weight models whose internals can be examined—the researchers used multiple-choice questions to categorize responses into four groups: correct and confident, correct and uncertain, incorrect and confident, and incorrect and uncertain.

Using sparse autoencoders to analyze internal activity, they identified three types of features: those linked primarily to uncertainty, those associated mainly with incorrect answers, and "confounded" features tied to both uncertainty and incorrectness.

Turning the knobs

The team then suppressed selected features while models answered questions. Patel compared this to "turning knobs" inside a model to observe behavioral changes.

The results revealed unexpected patterns. Disabling features associated purely with uncertainty sharply reduced accuracy, suggesting these features contribute to producing correct answers. Suppressing features linked only to incorrect responses had minimal effect.

The confounded features produced the most striking outcome. Suppressing features associated with both uncertainty and incorrectness improved accuracy by up to 1.1 percent while reducing expressed uncertainty by up to 75 percent. These interventions occurred during inference—when the trained model answers questions—requiring no retraining.

Teaching models when not to answer

In another experiment, researchers used just three confounded features from a single middle layer of the Llama model to predict when it would answer incorrectly. Having the model decline to answer flagged questions raised accuracy from 62 percent to 81 percent while still responding to about 53 percent of queries.

By contrast, simply giving the model an "I don't know" option and letting it self-select when to abstain raised accuracy only to 64 percent.

The identified features also generalized across different question-answering benchmarks, suggesting they reflect fundamental model characteristics rather than task-specific quirks.

Broader applications

Patel noted the approach could extend to other AI behaviors beyond confidence and correctness. Researchers could identify internal features associated with any desirable or problematic behavior, then manipulate those features to encourage or suppress specific outputs.

The study, "Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders," has been accepted for presentation at the SIAM International Conference on Data Mining in November. Co-authors include UCR Professor Evangelos E. Papalexakis, UCR Associate Professor Jia Chen, and Arizona State University researchers Tiejin Chen and Hua Wei. The findings were first reported by UC Riverside News.