A revised arXiv paper proposes an adaptive evaluation method that applies computerized adaptive testing ideas to generation tasks, where model outputs receive continuous scores instead of binary or multiple-choice marks. The approach aims to reduce the number of items needed while maintaining confident ranking of models. It targets LLM benchmarking beyond traditional multiple-choice setups.
A new arXiv paper introduces SALUTE, a benchmark designed to test how well large language models handle defense-related material, which relies on specialized terminology, doctrinal concepts and operational procedures. The authors also describe methods for adapting existing models to this domain, where military events and terminology shift over time. The work aims to measure and improve LLM performance in a knowledge-intensive field that general-purpose models often handle poorly.
Researchers have released MetroLLM-Bench, a set of 955 test cases that measures how well language models can act as the decision-making layer of a metro station kiosk. The benchmark draws on six real subway networks of varying sizes and covers eleven task types, including route planning, fare computation, and responding to service disruptions. The paper was posted to arXiv and cross-listed in the AI, computation and language, and machine learning categories.
A new arXiv paper introduces Φ-Bench, a benchmark that measures how well large language models can help develop and optimize the computing infrastructure used to run AI systems. The authors argue that existing benchmarks do not adequately cover these infrastructure-engineering tasks, which go beyond typical code generation. The work aims to gauge whether LLMs can realistically contribute to the specialized systems engineering that underpins their own operation.