AI coding agents ace isolated tests but complete fewer than one in ten tasks in simulated workplace
A new experiment places AI coding agents in a simulated software company where they must handle messy, ongoing development work. Even advanced models such as Claude Opus 5.5 and GPT-6 Astra fully complete fewer than one in ten tasks, despite high scores on standard benchmarks.

Key takeaways
- AI coding agents score highly on isolated benchmarks but complete fewer than one in ten tasks in a simulated workplace.
- Models including Claude Opus 5.5 and GPT-6 Astra show a significant drop in performance when acting autonomously.
- The gap between benchmark results and real-world workflow reliability remains large.
In this article
What the experiment found

The Decoder reports that researchers built a simulated developer job complete with colleagues, project boards and incomplete handoffs. AI coding agents were asked to keep developing software and close open tickets autonomously. While the models finished clear, standalone assignments quickly, they lost track when acting as autonomous developers over time.
Performance on benchmarks versus real work
Standard benchmarks such as SWE-bench are nearly saturated, and Anthropic says Claude writes more than 80 percent of its production code. However, the simulated workplace test shows a large gap between isolated benchmark success and the ability to manage a realistic, messy workflow.
Implications for autonomous development
The results suggest that current AI coding agents are not yet reliable enough to operate as independent digital employees. They excel at well-defined tasks but struggle with the context switching, incomplete information and long-term planning required in day-to-day software engineering.
Sources
Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.



