#calibration

13 curated events

papersTODAY 04:00 UTC

Paper Argues LLM-Judge Calibration in Biomedical ML Needs Four Separate Ledgers

A new arXiv paper examines how synthetic perturbations are often used as cheap calibration data for LLM evaluators in biomedical machine learning, where expert review is limited. The authors argue that a planted mutation key should not be treated as either a detector output or automatically as human ground truth. They propose formalizing four distinct ledgers to make reporting of calibration results more responsible.

llm-evaluation/biomedical/calibration/research

papersTODAY 04:00 UTC

Simulation-based inference used to calibrate ocean model mixing parameters

An arXiv preprint describes a method for tuning the free coefficients in vertical mixing parametrizations of single-column ocean models. Because these coefficients cannot be measured directly, the authors calibrate them against high-fidelity references such as large-eddy simulations using simulation-based inference. The approach is presented as an alternative to existing techniques that yield only point estimates.

research/simulation/ocean-modeling/calibration

papersTODAY 04:00 UTC

arXiv Paper Proposes Model-Agnostic Correctness Predictors for LLM Confidence

A new arXiv preprint introduces generalized correctness models, a method for predicting whether an LLM's answer is correct using patterns learned from historical behavior. The approach aims to produce confidence estimates that are both calibrated and transferable across different models. The authors frame the work as addressing the difficulty of obtaining reliable confidence signals for high-stakes or user-facing deployments.

calibration/llm-confidence/research/arxiv

papersTODAY 04:00 UTC

Calibeating generalized from quadratic scoring to all proper scoring rules

A new arXiv paper extends the concepts of calibrated forecasts and calibeating, which were previously defined only for the standard quadratic scoring rule, to the full class of proper scoring rules. The authors develop these notions in this broader setting, where truthful reporting is a defining property of the rule. The work aims to show how calibration-based guarantees carry over beyond the squared-error case.

calibration/forecasting/scoring-rules/research

papersTODAY 04:00 UTC

Agentic AI workflow automates TCAD calibration for oxide semiconductor transistors

Researchers present an agent-based workflow that automates experimental TCAD calibration for emerging oxide semiconductor transistors, a task currently done by hand and dependent on expert judgment. The approach targets the model ambiguity that arises when several physical models and parameter sets can fit the same measurements. The paper is posted on arXiv as a cross-listed replacement version.

agents/tcad/semiconductors/calibration

papersTODAY 04:00 UTC

Calibrated Uncertainty Estimation for LLM Clinical Text Classification

A new arXiv paper addresses the risk of overconfident errors when large language models classify clinical text, where a wrong label can affect patient care. The authors note that current black-box approaches simply attach a confidence score to an unchanged LLM prediction, and they propose an uncertainty-aware method designed to produce better-calibrated results. The work targets medical NLP settings where knowing when a model is unreliable matters as much as the predicted label itself.

clinical-nlp/uncertainty/calibration/llm

papersSEP 11 04:00 UTC

arXiv paper targets lower-tail calibration of Gaussian processes for Bayesian optimization

An updated arXiv preprint proposes a goal-oriented approach to calibrating the lower tail of Gaussian process predictive distributions, which Bayesian optimization uses to choose where to evaluate costly objective functions. The abstract notes that kernel and hyperparameter choices strongly shape these predictions. The submission is a replacement version (v2) of an earlier preprint.

bayesian-optimization/gaussian-processes/calibration/arxiv

papersSEP 10 04:00 UTC

Cost-Aware Deferral for Classifiers Under Calibration Shift: Environmental AI Case Study

A new arXiv preprint examines how to choose a deferral policy for a fixed classifier, where uncertain cases are routed to human reviewers. The authors analyze how miscalibrated confidence scores, unequal error costs, fallible reviewers, and deployment-time distribution shift interact, using an environmental AI application as a real-world case study. The work offers practical guidance for deciding when automated predictions should be handed off rather than trusted outright.

deferral/calibration/human-in-the-loop/environmental-ai

papersSEP 10 04:00 UTC

New arXiv paper proposes calibrating AI agent confidence from internal representations

A newly released arXiv paper addresses how to measure the confidence behind agentic AI actions, arguing this is essential as such systems enter safety-critical applications. The authors note that agentic workflows fail in more complex ways than traditional machine learning systems and propose deriving calibrated confidence estimates directly from the model's internal representations.

agents/safety/calibration/research

papersSEP 10 04:00 UTC

ProbPlug: A Plugin Network for Reliable Confidence Estimates in LLM Binary Classification

Researchers have introduced ProbPlug, a plugin uncertainty network designed to attach to large language models and produce more trustworthy confidence scores for binary classification tasks. The work addresses the gap between strong LLM predictive performance and the reliability required for deployment in high-stakes settings. The paper is available as a preprint on arXiv.

llm/calibration/uncertainty/research

papersSEP 10 04:00 UTC

Reliability-Aware Hybrid-K Ensemble Selection Proposed for Cervical Cytology Classification

A new arXiv preprint introduces a hybrid ensemble selection framework for multiclass cervical cytology image classification that weighs discriminative performance alongside calibration and selective prediction. The authors argue that raw accuracy is not enough for clinical image analysis, and that the method is meant to deliver trustworthy confidence scores and uncertainty flags so users know when to distrust a prediction. The work appears in the cs.AI category as an early-stage research contribution.

medical-ai/computer-vision/calibration/uncertainty

papersSEP 12 04:00 UTC

Calibration Audit Questions Confidence Scores in Feed-Forward 3D Reconstruction Models

A study examines whether the per-pixel confidence values produced by feed-forward 3D reconstruction models can be treated as reliable uncertainty estimates. Because these scores are trained mainly as loss weights, the authors audit how well they are calibrated for downstream use. The paper reports on the limits of reusing them as an uncertainty signal.

3d-reconstruction/uncertainty/calibration/arxiv

papersSEP 12 04:00 UTC

Study Finds Confidence Scale Design Shapes LLM Uncertainty Estimates

A new arXiv paper examines how the numerical scale used for verbalized confidence, usually 0-100, affects the reliability of LLM uncertainty estimates. The authors argue this design choice is not neutral and that scale design should be treated as a meaningful variable in black-box calibration research.

llm/calibration/uncertainty/research