Skip to content
Theme
OnDevice Tools

AIChatbots & Assistants

AI coding agents ace isolated tests but complete fewer than one in ten tasks in simulated workplace

A new experiment places AI coding agents in a simulated software company where they must handle messy, ongoing development work. Even advanced models such as Claude Opus 5.5 and GPT-6 Astra fully complete fewer than one in ten tasks, despite high scores on standard benchmarks.

Illustration of AI coding agents working in a simulated software company environment

Key takeaways

  • AI coding agents score highly on isolated benchmarks but complete fewer than one in ten tasks in a simulated workplace.
  • Models including Claude Opus 5.5 and GPT-6 Astra show a significant drop in performance when acting autonomously.
  • The gap between benchmark results and real-world workflow reliability remains large.
In this article
  1. What the experiment found
  2. Performance on benchmarks versus real work
  3. Implications for autonomous development

What the experiment found

Illustration: code, AI, Claude logo, performance and the labels AI Coding Agents, Chatbots & Assistants

The Decoder reports that researchers built a simulated developer job complete with colleagues, project boards and incomplete handoffs. AI coding agents were asked to keep developing software and close open tickets autonomously. While the models finished clear, standalone assignments quickly, they lost track when acting as autonomous developers over time.

Performance on benchmarks versus real work

Standard benchmarks such as SWE-bench are nearly saturated, and Anthropic says Claude writes more than 80 percent of its production code. However, the simulated workplace test shows a large gap between isolated benchmark success and the ability to manage a realistic, messy workflow.

Implications for autonomous development

The results suggest that current AI coding agents are not yet reliable enough to operate as independent digital employees. They excel at well-defined tasks but struggle with the context switching, incomplete information and long-term planning required in day-to-day software engineering.

Sources

  1. AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplaceThe Decoder

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.