Science

AI models excel at coding but struggle with judgment tasks

METR's updated evaluation framework reveals a growing gap between AI performance on verifiable problems and open-ended work requiring research taste.

Omega Editorial · November 20, 2026 · 4 min read

Key takeaways

  • METR's updated evaluation shows AI models continue exponential improvement on verifiable tasks like coding but lag significantly on judgment-heavy work
  • Frontier models failed to design a single AI evaluation useful enough for METR researchers to actually use, despite excelling at software engineering
  • The performance gap matters for predicting autonomous AI research and development, which requires research taste and judgment that current systems lack
  • Newest models show incremental progress on open-ended tasks, moving from 'clearly below' to 'borderline above' the bar in one year
  • Evaluating judgment and open-ended capabilities remains methodologically difficult, making it hard to predict whether and when models will close the gap

AI systems continue to advance at an exponential pace on tasks with clear right-or-wrong answers, but a new evaluation from the nonprofit METR reveals they still fall short when judgment and research intuition are required.

The organization has updated its widely-cited time horizon plot, which tracks how AI capabilities scale with task complexity. The original framework measured models against a suite of tasks that humans had completed, scoring difficulty by how long those tasks took human testers. Models were then rated on the longest tasks they could successfully finish. That data showed frontier models doubling their capability range every four months.

The new evaluation adds significantly harder tasks—some estimated to take humans hundreds of hours—and crucially, more open-ended assignments where success is harder to define automatically.

The verification gap

The distinction matters because of how modern AI systems learn. Reinforcement learning works well when researchers can automatically check whether an answer is correct, which is why models have become exceptionally good at coding and mathematics. Neural networks attempt these tasks repeatedly, strengthening pathways that lead to success and weakening those that produce failures.

But many real-world problems—particularly in research and development—resist this approach. Choosing a promising research direction or designing a useful experiment requires judgment that can't be verified automatically. These are precisely the capabilities needed for AI systems to meaningfully accelerate scientific progress or, more controversially, to improve themselves autonomously.

METR's expanded test suite now includes tasks designed to probe this judgment capacity. In one evaluation, models must design new AI tests that METR researchers could actually use. Despite frontier models now outperforming most humans at software engineering, not a single AI-designed evaluation met METR's bar for usefulness.

In another task measuring philosophical reasoning—intended as a proxy for good judgment—models again underperformed humans, though METR notes internal disagreement about how relevant this particular test is for predicting R&D automation.

Why it matters

The performance gap on judgment-heavy tasks has direct implications for near-term AI deployment and longer-term safety considerations. Companies are racing to build AI systems that can conduct autonomous research, and a recent METR report suggested AI might already be doubling the pace of capabilities progress at Anthropic. Understanding whether models can transfer their strength on verifiable tasks to open-ended problems is essential for predicting when—or if—truly autonomous AI R&D becomes possible.

The challenge extends beyond measurement. Sayash Kapoor, an incoming UC Berkeley professor who has studied AI's ability to reproduce computer science research, told MIT Technology Review he wasn't surprised by the results. His August preprint found models cannot yet independently replicate human research. OpenAI reported similar findings this week, noting its models perform worse on real employee tasks than on easier benchmark evaluations.

Incremental progress, uncertain trajectory

There are signs of improvement. The newest models, including Opus 5.5, achieve what METR calls "borderline above the bar" performance on evaluation design—not yet usable but more promising than the "clearly below the bar" scores from a year ago.

But predicting future progress remains difficult. The new evaluations rely heavily on estimated completion times rather than measured ones, since having humans complete hundred-hour tasks is prohibitively expensive. Some tasks, like the philosophical reasoning test, were scored by a single expert, raising questions about generalizability.

Matt Putz, a METR technical staff member, acknowledged these limitations in an email to MIT Technology Review, noting that "by their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify."

Boston University computer scientist Najoung Kim said the current results provide a useful snapshot but cautioned against extrapolation: "I'm not sure just yet how to think about how it would extrapolate."

The debate over whether models trained on verifiable tasks will eventually master open-ended ones remains unresolved. Progress on coding and math shows no signs of slowing, and some researchers believe those capabilities will transfer. Others, including Kapoor, argue that research taste—the ability to identify promising questions and approaches—will prove a fundamental bottleneck.

The findings were first reported by MIT Technology Review.

  • #ai evaluation
  • #metr
  • #ai capabilities
  • #reinforcement learning
  • #autonomous research
  • #ai benchmarks

This is an original analysis by the Omega editorial team. Source reporting: AI Watch.

Want systems like this working for your business?

Book a Call

More in Science