METR updates AI time horizon plot as models lag on judgment tasks
METR has released an updated version of its AI time horizon plot, adding harder and more open-ended tasks. The report shows AI systems still improve quickly on verifiable tasks such as coding but fall short of humans on tasks that require judgment.

Key takeaways
- METR has updated its AI time horizon plot with harder, more open-ended tasks.
- AI models continue to improve quickly on verifiable tasks such as coding and math.
- Models still fall short of humans on judgment-heavy tasks such as designing AI evaluations.
- Few open-ended tasks are included because they are expensive and difficult to score.
In this article
What the updated report found
METR, an AI evaluation nonprofit, has published an update to its time horizon plot. The plot measures AI capability by how long a task takes a human to complete and then rates models by the length of tasks they can finish. MIT Technology Review AI reports that frontier models continue to improve along an exponential path, with the length of tasks they can complete doubling roughly every four months.
Judgment tasks remain a weakness
The new report adds tasks that are more open-ended and harder to verify, including designing AI evaluations and scoring philosophical arguments. On these tasks, even the most advanced models fall short of human performance. None of the AI-designed evaluations were considered useful enough by METR staff for their own research.
Why verifiable tasks are easier to improve
Reinforcement learning works well for verifiable tasks such as coding and math because success can be checked automatically. It is much harder to apply to tasks where success is poorly defined or takes long to judge, such as choosing a new research direction. This gap matters because research and development tasks will be essential if AI systems are to automate AI progress.
Context and recent debate

The update follows a heavily redacted METR report from earlier this month that claimed AI might be speeding up capabilities progress at Anthropic by a factor of two, sparking debate about recursive self-improvement. Anthropic recently completed a blockbuster IPO. OpenAI has also said its models perform worse on real-world employee tasks than on easier verifiable tasks in METR's original suite.
What experts say
Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to get more creative and that pushing the envelope of evaluation methods is welcome. He was not surprised that models lag on open-ended tasks. Najoung Kim, a linguist and computer scientist at Boston University, said the observed improvement tells something about the current moment but it is unclear how to extrapolate it. Matt Putz of METR said hard-to-verify tasks are much harder to study and quantify, and clean trends may not be possible.
Limitations of the new data
METR includes few open-ended tasks in the report because they are difficult to design and score. In the original plot, human testers completed each task to measure completion time. In the new report, most completion times are estimates because paying people to complete tasks that could take hundreds of hours is prohibitively expensive. METR says this makes it difficult to predict how models might progress on judgment-heavy tasks.
Sources
- The most misunderstood graph in AI just got a major updateMIT Technology Review AI
Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.



