A research study by Shandong University scholars indicates that complex agent harnesses—such as Claude Code and OpenCode—do not significantly improve coding LLM accuracy compared to lightweight setups, despite drastically increasing token consumption and cost. Evaluating tools on the SWE-bench Verified benchmark, researchers found that performance variations between different harnesses fall within the statistical noise of standard re-runs.
The study evaluated five LLMs across three harness architectures ranging from basic CLI setups to tool-heavy frameworks with context compression. On a benchmark set of 447 coding tasks, heavy harnesses performed within two points of minimal configurations like mini-SWE-agent. Furthermore, switching harnesses yielded identical task-flip rates (13%) as simply re-running the exact same model setup a second time.
The primary measurable distinction between harnesses was cost, with detailed context management driving token usage up to three times higher. The researchers also noted that web-enabled evaluations often skewed past leaderboard metrics due to models retrieving direct GitHub solutions, emphasizing the need for isolated offline benchmarking.
Why it matters
Warns engineering leads against over-spending on complex commercial agent harnesses that increase API bills without improving benchmark accuracy.
Highlights the prevalence of high variance and noise in popular agent benchmarks like SWE-bench, urging teams to run multi-pass evaluations.
Suggests developer tool builders should focus on lightweight harness designs to minimize latency and token consumption.
Source: the-decoder.com



