A new arXiv paper introduces IWC-Bench, a benchmark that assesses LLM-generated web applications from a software testing angle. The authors argue that existing static benchmarks can reward functionality that appears in source code but does not actually work when the app runs. Their approach aims to make automated evaluation of generated web apps track human judgments more closely.
Researchers introduce Vibe Patenting, a testbed that evaluates whether LLM judges can reliably assess AI agents performing professional patent-drafting work. The work probes how dependable automated evaluation is when applied to complex, specialized tasks rather than general benchmarks. It highlights open questions about using LLMs as evaluators in high-stakes professional settings.
A new arXiv paper examines how well automated evaluators judge whether jailbreak attacks on language models succeed, noting that human expert review is expensive and hard to scale. The authors argue that jailbreak research often fails to properly validate the evaluators it depends on, and they empirically measure those evaluators' behavior.
A new arXiv paper introduces XAI-Arena, a study of whether large language models can reliably evaluate explanations produced by explainable AI methods. The authors note that current evaluation relies heavily on subjective human judgment, which hurts reproducibility, scalability, and comparability across studies. The work explores automated, LLM-based assessment as a potential alternative to manual expert reviews.