Researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Tech, M-A-P, and TokenWave.AI have introduced HarnessDev, a evaluation framework designed to test whether LLMs can engineer their own executable agent harnesses. The benchmark evaluates generated harnesses across capability and efficiency using 2,207 instances across five domains, testing creator LLMs including Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro.
The study found that while models can improve their harnesses through iterative feedback, much of the generated code remains brittle or non-functional. Under Self-Eval, Opus 4.8 achieved the highest average capability score at 67.8, compared to a human-engineered reference score of 86.2. However, code volume did not correlate with performance, and out of 108 generated code component instances, 18 state and memory features never executed during testing.
Crucially, agent harnesses suffered significant performance degradation when swapped to different underlying executor models. For example, changing the executor to Gemini 3.1 Pro reduced Opus 4.8’s SWE-Pro score from 69.3 to 33.0 due to hard-coded assumptions like rigid step limits, demonstrating that current LLMs struggle to write generalizable, portable agent infrastructure.
Why it matters
Provides critical empirical data for engineers building agentic scaffolds, showing human-designed harnesses still vastly outperform LLM-generated code.
Highlights severe portability issues when decoupling model orchestrators from specific underlying LLM executors.
Exposes significant inefficiency and dead code risks in self-evolving or auto-generated software engineering pipelines.
Source: marktechpost.com
