Researchers present CALICO, a human-centered system that helps domain experts turn their annotation codebooks into prompts for large language models. The work targets gaps in existing pipelines, which offer little support for producing prompts that stay reliable, easy to revise, and auditable. It is described in a paper posted to arXiv under the cs.CL category.
Researchers present CoTAL, a human-in-the-loop prompt engineering method for using large language models to score formative assessments and generate feedback for students. The work examines how well such prompting approaches generalize across educational contexts, with teachers involved in refining the prompts. It is published as an arXiv preprint in the computation and language category.
A new arXiv paper presents a human-in-the-loop meta Bayesian optimization framework aimed at experiments where each trial is costly and scarce, such as inertial confinement fusion. The approach combines learned meta-level priors with human feedback to guide the search over experimental parameters under tight budget constraints. The authors frame the method as applicable to scientific domains beyond fusion that face similar cost and access limits.
A research paper addresses how to decide which LLM-judged outputs actually need human expert review. It separates the judge's uncertainty into aleatoric uncertainty, which reflects genuine disagreement among experts and cannot be reduced by more labels, and epistemic uncertainty, which signals where expert annotation would help. The goal is to spend limited expert labeling effort on the cases where it is most useful.
A new arXiv preprint examines how to choose a deferral policy for a fixed classifier, where uncertain cases are routed to human reviewers. The authors analyze how miscalibrated confidence scores, unequal error costs, fallible reviewers, and deployment-time distribution shift interact, using an environmental AI application as a real-world case study. The work offers practical guidance for deciding when automated predictions should be handed off rather than trusted outright.