A new arXiv paper identifies a vulnerability in evaluation pipelines that use LLM-based judges guided by natural-language rubrics. The authors show that rubrics can serve as an attack surface, allowing subtle preference drift in judge behavior that may go unnoticed by standard benchmarks. The work highlights the need for more robust validation of rubric-driven evaluation and alignment setups.
Researchers introduce Vibe Patenting, a testbed that evaluates whether LLM judges can reliably assess AI agents performing professional patent-drafting work. The work probes how dependable automated evaluation is when applied to complex, specialized tasks rather than general benchmarks. It highlights open questions about using LLMs as evaluators in high-stakes professional settings.
A new arXiv paper examines large language models acting as evaluators under rubric-based protocols, a setting that has received less attention than pointwise and pairwise comparison methods. The authors find that the ordering of responses systematically influences the judge's verdicts, exposing position bias in this evaluation setup. The results suggest that pipelines relying on LLM judges may need safeguards or reordering strategies to produce reliable assessments.
A research paper on arXiv introduces a method for scoring text against evaluation rubrics using large language models, addressing how black-box judge models can apply identical criteria in inconsistent ways. The approach ties each score to concrete evidence drawn from the evaluated text, making the reasoning behind judgments easier to audit and reproduce. The work is cross-listed under arXiv categories for artificial intelligence, computational linguistics, and machine learning.