Researchers from Microsoft and the City University of Hong Kong have released Taste-Bench, a benchmark designed to evaluate how effectively AI agents make choices during long-running tasks. Testing 14 major models across 502 binary decision scenarios, the researchers found that even the top-performing model, GPT-5.6 Sol, achieved an accuracy of only 59.7 percent when selecting the superior trajectory at critical decision forks.
Existing benchmarks typically evaluate whether an AI agent completes a task, ignoring the quality of intermediate decisions whose costs only become apparent later. Taste-Bench addresses this gap by freezing agent execution at decision points, hiding subsequent work, and requiring models to choose between two potential paths derived from coding and AI research trajectories.
The findings highlight a major limitation in current autonomous systems, as average agent performance dropped to 21 percent when key decision clues only appeared after extensive downstream work. The authors note that developing AI with better decision-making “taste” will be critical as agents are increasingly deployed for complex, multi-step software development and research projects.
Why it matters
AI agents struggle with long-horizon decision-making, leading to wasted compute and budget on flawed execution paths.
Benchmarks like Taste-Bench provide essential evaluation tools for teams building autonomous software engineering and research agents.
Source: the-decoder.com



