Skip to content
Theme
OnDevice Tools

AIAI Models & Research

METR updates AI time horizon plot as models lag on judgment tasks

METR has released an updated version of its AI time horizon plot, adding harder and more open-ended tasks. The report shows AI systems still improve quickly on verifiable tasks such as coding but fall short of humans on tasks that require judgment.

Illustration of a graph showing AI task completion time improving over time

Key takeaways

  • METR has updated its AI time horizon plot with harder, more open-ended tasks.
  • AI models continue to improve quickly on verifiable tasks such as coding and math.
  • Models still fall short of humans on judgment-heavy tasks such as designing AI evaluations.
  • Few open-ended tasks are included because they are expensive and difficult to score.
In this article
  1. What the updated report found
  2. Judgment tasks remain a weakness
  3. Why verifiable tasks are easier to improve
  4. Context and recent debate
  5. What experts say
  6. Limitations of the new data

What the updated report found

METR, an AI evaluation nonprofit, has published an update to its time horizon plot. The plot measures AI capability by how long a task takes a human to complete and then rates models by the length of tasks they can finish. MIT Technology Review AI reports that frontier models continue to improve along an exponential path, with the length of tasks they can complete doubling roughly every four months.

Judgment tasks remain a weakness

The new report adds tasks that are more open-ended and harder to verify, including designing AI evaluations and scoring philosophical arguments. On these tasks, even the most advanced models fall short of human performance. None of the AI-designed evaluations were considered useful enough by METR staff for their own research.

Why verifiable tasks are easier to improve

Reinforcement learning works well for verifiable tasks such as coding and math because success can be checked automatically. It is much harder to apply to tasks where success is poorly defined or takes long to judge, such as choosing a new research direction. This gap matters because research and development tasks will be essential if AI systems are to automate AI progress.

Context and recent debate

Illustration: Anthropic logo, AI, update, code and the labels AI Evaluation, Metr, Reinforcement Learning

The update follows a heavily redacted METR report from earlier this month that claimed AI might be speeding up capabilities progress at Anthropic by a factor of two, sparking debate about recursive self-improvement. Anthropic recently completed a blockbuster IPO. OpenAI has also said its models perform worse on real-world employee tasks than on easier verifiable tasks in METR's original suite.

What experts say

Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to get more creative and that pushing the envelope of evaluation methods is welcome. He was not surprised that models lag on open-ended tasks. Najoung Kim, a linguist and computer scientist at Boston University, said the observed improvement tells something about the current moment but it is unclear how to extrapolate it. Matt Putz of METR said hard-to-verify tasks are much harder to study and quantify, and clean trends may not be possible.

Limitations of the new data

METR includes few open-ended tasks in the report because they are difficult to design and score. In the original plot, human testers completed each task to measure completion time. In the new report, most completion times are estimates because paying people to complete tasks that could take hundreds of hours is prohibitively expensive. METR says this makes it difficult to predict how models might progress on judgment-heavy tasks.

Sources

  1. The most misunderstood graph in AI just got a major updateMIT Technology Review AI

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.