A new arXiv paper reviews how large language models used in clinical and medical settings should be assessed. It argues that evaluating these systems is harder than conventional machine learning evaluation for a variety of reasons. The work is framed as an introductory guide to evaluation approaches for healthcare LLMs.
A new arXiv preprint introduces Asclepius, a harness designed to keep LLM agents stable during long clinical tasks rather than short single-step prompts. The authors evaluate it in a Clinical Environment Simulator where an agent handles extended workflows under resource contention, arguing that current benchmarks miss this class of failures. The work targets reliability problems that appear only in hours-long deployments.
A new preprint examines how correctness is defined when vision-language models are used to make clinical predictions from medical images and electronic health records. The authors argue that current uncertainty estimation methods may be evaluated in ways that do not reflect whether a prediction is actually reliable. The work targets safer deployment by improving how unreliable outputs are detected.
A new arXiv paper examines how clinical LLM agents behave when given the same patient case multiple times. Although the agents often reach the same overall judgment, the tests, medications, and referrals they order can differ substantially between runs. The authors argue that evaluating these agents on a single run per task can hide this variability and misrepresent their reliability.
A new arXiv paper proposes updating intensive care prediction models through a structurally bounded procedure that leaves the physiological encoder frozen. The authors argue this limits how much model behavior and its explanations can drift when patient data distributions change. The aim is to make adapted clinical models easier to audit after deployment.
Researchers present EasyLens, a plug-and-play method that amplifies the representation of subtle lesions in medical vision-language models without requiring any additional training. The approach targets a known weakness of such models, whose clinical usefulness is limited by low sensitivity to faint or small abnormalities. According to the paper, the technique can be added to existing pipelines to improve lesion detection and related report generation tasks.
A new arXiv paper addresses the stopping problem for AI agents in clinical diagnosis, which must decide when to request another test, when to commit to a diagnosis, and when to defer. The authors note that current agent benchmarks typically measure accuracy under fixed or unconstrained interaction, leaving the reliability of autonomous stopping untested. They propose a risk-constrained framework for evaluating and controlling these stopping decisions in sequential diagnosis settings.