Researchers have introduced ThinkingBox, a benchmark that evaluates AI agents across 507 stateful business workflows by measuring backend database states and side effects rather than relying on final text responses or tool-call syntax. Evaluated across 12 major large language models over 121,680 test trials, the benchmark revealed significant gaps between reported task completions and actual database accuracy.
According to the published findings, 67.24% of failed attempts terminated cleanly and reported no tool errors despite failing executable state checks. Common underlying backend errors included incorrect database field values (77.61%), unintended extra side effects (43.30%), and missing required state updates (25.36%). The benchmark requires agents to pass 20 repeated trials from identical clean backends to establish reliable performance.
Why it matters
Demonstrates that standard evaluation metrics like pass@1 and valid tool-call traces significantly overestimate autonomous agent reliability in enterprise workflows.
Provides developers with an open benchmark methodology for testing database side effects and execution consistency across LLM function calling.
Source: huggingface.co



