Paper Argues Policy Ambiguity Skews Agent Benchmark Results

A new arXiv paper contends that agent benchmarks assume each policy implies one correct action, an assumption that natural-language policies often break through silence, ambiguity, or contradiction. The authors describe these as policy loopholes, where multiple defensible readings exist but evaluations still count a single behavior as an agent error. The work suggests such ambiguous cases should be separated from genuine policy-compliance failures in benchmark scoring.