A new arXiv preprint argues that identifying why long-horizon AI agents fail is best treated as a search problem over large execution logs. The approach aims to turn outcome-level failure signals into targeted fixes by locating the specific steps that caused a breakdown. The work targets reliability engineering for agents deployed on extended, multi-step tasks.
A new arXiv paper introduces the concept of "efficiency hallucination," describing how large language models sometimes propose code changes that do not alter functionality while claiming performance gains that are not actually supported. The authors formalize and measure this behavioral calibration problem to assess how reliably LLMs can justify their optimizations. The work aims to give researchers a way to quantify when model claims about speedups diverge from verified results.
A new arXiv paper examines why generate-test-revise loops in coding agents fail to guarantee dependable code repair, focusing on the gap between producing a correct patch and keeping, verifying, and submitting it. The authors propose state-bound evidence and typed revision contracts to make the revision process more accountable. Their sealed five-seed study covers 30 tasks.
A new arXiv paper examines the use of local large language models as automated judges of other models' outputs, a practice meant to cut the cost and time of human evaluation. The authors investigate whether a judge that returns stable, consistent scores can still be unreliable when compared against human ratings. The work argues that consistency alone is not sufficient evidence of a trustworthy evaluator.
A new arXiv paper argues that long-running AI agents interrupted mid-task need a built-in notion of recoverability, since restarting wastes effort while resuming from unverified state can propagate earlier mistakes. The authors frame saved state alone as insufficient and propose treating recovery as a core system-level capability rather than an afterthought. The work is a preprint and has not yet been peer reviewed.
A new arXiv paper examines how clinical LLM agents behave when given the same patient case multiple times. Although the agents often reach the same overall judgment, the tests, medications, and referrals they order can differ substantially between runs. The authors argue that evaluating these agents on a single run per task can hide this variability and misrepresent their reliability.
A new arXiv paper examines how AI agents that run long workflows through external tools can leave inconsistent state even when individual tool calls report success. It focuses on conditions such as retries, speculative execution, concurrency, and partial failures. The work frames these mismatches as anomalies at the boundary between the agent and the tools it calls.
A new arXiv paper proposes RESKILL, a method for repairing language agents that relies on explicit failure attribution instead of opaque one-shot reflection. The approach keeps a structured link between explanations of why a task failed and the candidate skill patches meant to fix it, rather than generating a patch in a single unexamined pass. The authors argue this makes post-failure repair more transparent and easier to verify.
A Hacker News discussion examines the practice of using one large language model to grade another's output, and asks whether consensus among several such judges actually indicates a correct verdict. Commenters raise concerns that models can share the same blind spots or biases, so agreement may reflect correlated error rather than genuine quality. The thread touches on how evaluation setups should be validated, for example against human raters or adversarial examples.
Researchers propose Referee-Based Quality Estimation, a framework that scores polyp segmentation output without ground-truth labels by measuring how much a primary model agrees with other models. The approach is intended as a deployment-time signal that catches silent failures during real-time colonoscopy, when annotations are not available at inference. The work is published as an arXiv preprint.
Researchers assembled a corpus of 150 academic papers with deliberately planted errors to test how well large language models can act as automated document-quality auditors. They report that detection reliability worsens as processing batch sizes increase, and that models sometimes fabricate audit findings with high confidence. The results cast doubt on deploying LLMs unsupervised for contamination-detection tasks.
A new arXiv paper investigates whether iterative self-refinement can make language model agents that interact with tools through code more reliable at inference time. The authors show that the benefits of refinement depend heavily on the feedback format, with unstructured critique producing inconsistent results across models. They introduce RubricRefine, a training-free method that refines agent outputs before execution using rubric-based feedback.
A new arXiv preprint addresses the tendency of large language models to make arithmetic mistakes, which is risky in clinical calculators where an error can change a care recommendation. The authors argue against manually hardcoding each calculator as a separate validated function and instead propose a deterministic math solving approach for clinical language models. The abstract is truncated, so full details of the method and evaluation are not yet available.