A research team from Alibaba and several Asian universities has introduced Co-Design Bench, a new coding agent benchmark derived from real user change requests. The study found that even the top-performing model, GPT-6 Astra, achieved completely correct results in only 26.4 percent of tasks, requiring human developers to clean up nearly three out of four assignments.

The authors created the benchmark to address the saturation of older tests like SWE-bench Verified, where self-correcting agents frequently score above 99 percent. Co-Design Bench tests models on multi-modal inputs—including video, audio, images, and documents—to modify existing web applications. While GPT-6 Astra fulfilled roughly 70 percent of task requirements on average, it routinely missed key implementation details on complex requests.

The study also highlighted that working environments significantly alter agent performance. A model’s success rate swung by up to 19 percentage points depending on the software harness used, leading researchers to recommend that companies evaluate model and environment pairings together rather than relying on isolated model leaderboards.

Why it matters

  • Highlights the gap between high benchmark scores and real-world utility, where 74% of tasks still require human intervention.

  • Demonstrates that harness choice impacts performance as heavily as model selection, requiring joint system testing.

  • Shows extra compute yields diminishing returns on complex multi-requirement coding overhauls.

Source: the-decoder.com