Science
AI models excel at coding but struggle with judgment tasks
METR's updated evaluation framework reveals a growing gap between AI performance on verifiable problems and open-ended work requiring research taste.
Key takeaways
- METR's updated evaluation shows AI models continue exponential improvement on verifiable tasks like coding but lag significantly on judgment-heavy work
- Frontier models failed to design a single AI evaluation useful enough for METR researchers to actually use, despite excelling at software engineering
- The performance gap matters for predicting autonomous AI research and development, which requires research taste and judgment that current systems lack
- Newest models show incremental progress on open-ended tasks, moving from 'clearly below' to 'borderline above' the bar in one year
- Evaluating judgment and open-ended capabilities remains methodologically difficult, making it hard to predict whether and when models will close the gap
AI systems continue to advance at an exponential pace on tasks with clear right-or-wrong answers, but a new evaluation from the nonprofit METR reveals they still fall short when judgment and research intuition are required.
The organization has updated its widely-cited time horizon plot, which tracks how AI capabilities scale with task complexity. The original framework measured models against a suite of tasks that humans had completed, scoring difficulty by how long those tasks took human testers. Models were then rated on the longest tasks they could successfully finish. That data showed frontier models doubling their capability range every four months.
The new evaluation adds significantly harder tasks—some estimated to take humans hundreds of hours—and crucially, more open-ended assignments where success is harder to define automatically.
The verification gap
The distinction matters because of how modern AI systems learn. Reinforcement learning works well when researchers can automatically check whether an answer is correct, which is why models have become exceptionally good at coding and mathematics. Neural networks attempt these tasks repeatedly, strengthening pathways that lead to success and weakening those that produce failures.
But many real-world problems—particularly in research and development—resist this approach. Choosing a promising research direction or designing a useful experiment requires judgment that can't be verified automatically. These are precisely the capabilities needed for AI systems to meaningfully accelerate scientific progress or, more controversially, to improve themselves autonomously.
METR's expanded test suite now includes tasks designed to probe this judgment capacity. In one evaluation, models must design new AI tests that METR researchers could actually use. Despite frontier models now outperforming most humans at software engineering, not a single AI-designed evaluation met METR's bar for usefulness.
In another task measuring philosophical reasoning—intended as a proxy for good judgment—models again underperformed humans, though METR notes internal disagreement about how relevant this particular test is for predicting R&D automation.
Why it matters
The performance gap on judgment-heavy tasks has direct implications for near-term AI deployment and longer-term safety considerations. Companies are racing to build AI systems that can conduct autonomous research, and a recent METR report suggested AI might already be doubling the pace of capabilities progress at Anthropic. Understanding whether models can transfer their strength on verifiable tasks to open-ended problems is essential for predicting when—or if—truly autonomous AI R&D becomes possible.
The challenge extends beyond measurement. Sayash Kapoor, an incoming UC Berkeley professor who has studied AI's ability to reproduce computer science research, told MIT Technology Review he wasn't surprised by the results. His August preprint found models cannot yet independently replicate human research. OpenAI reported similar findings this week, noting its models perform worse on real employee tasks than on easier benchmark evaluations.
Incremental progress, uncertain trajectory
There are signs of improvement. The newest models, including Opus 5.5, achieve what METR calls "borderline above the bar" performance on evaluation design—not yet usable but more promising than the "clearly below the bar" scores from a year ago.
But predicting future progress remains difficult. The new evaluations rely heavily on estimated completion times rather than measured ones, since having humans complete hundred-hour tasks is prohibitively expensive. Some tasks, like the philosophical reasoning test, were scored by a single expert, raising questions about generalizability.
Matt Putz, a METR technical staff member, acknowledged these limitations in an email to MIT Technology Review, noting that "by their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify."
Boston University computer scientist Najoung Kim said the current results provide a useful snapshot but cautioned against extrapolation: "I'm not sure just yet how to think about how it would extrapolate."
The debate over whether models trained on verifiable tasks will eventually master open-ended ones remains unresolved. Progress on coding and math shows no signs of slowing, and some researchers believe those capabilities will transfer. Others, including Kapoor, argue that research taste—the ability to identify promising questions and approaches—will prove a fundamental bottleneck.
The findings were first reported by MIT Technology Review.