AI agent benchmarks can be slashed by ~50% in cost by only evaluating on tasks with intermediate historical pass rates.
March 26, 2026
Original Paper
Efficient Benchmarking of AI Agents
arXiv · 2603.23749
The Takeaway
Interactive agent evaluations are prohibitively expensive. This paper proves that rank-order stability is preserved even when discarding nearly half the benchmark tasks, provided you select for 'mid-range difficulty' based on Item Response Theory, making leaderboards much more sustainable to maintain.
From the abstract
Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurat