LIVE PULSE
4.9 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.6 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.4 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.7 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.3 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.3 CoMem Paper Proposes Shared and Individual Memory Design for LLM Multi-Agent Systems — 1 src1.3 Paper proposes evolving context parameterization for large language models — 1 src1.3 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src4.9 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.6 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.4 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.7 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.3 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.3 CoMem Paper Proposes Shared and Individual Memory Design for LLM Multi-Agent Systems — 1 src1.3 Paper proposes evolving context parameterization for large language models — 1 src1.3 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up

#llm-as-judge

3 curated events
papersTODAY 04:00 UTC

GAVEL: LLM Judge Protocol for Comparing Extracted Clinical Timelines Against Case Reports

Researchers introduce GAVEL, a protocol that uses a large language model as a judge to compare two clinical timelines extracted from case reports, rather than relying on a single expert reference annotation. The approach aims to address limitations in existing extraction pipelines, where evaluation is constrained by imperfect reference labels and imprecise event alignment. The work is described in a new arXiv preprint.

llm-as-judge/clinical-nlp/evaluation/arxiv
papersTODAY 04:00 UTC

Study Finds LLM Judges Underuse Non-Directional Verdicts Allowed by Task Contracts

A new arXiv paper examines how large language models act as judges in evidence-based fact verification, converting supporting material into final verdicts. The authors report that even when task instructions explicitly permit non-directional outcomes such as "Conflicting" or "Not Enough Evidence," models tend to favor directional verdicts instead. The work suggests a mismatch between stated judging criteria and the labels models actually produce.

llm-as-judge/evaluation/fact-checking/research
papersYESTERDAY 16:29 UTC

Hacker News thread debates whether agreement between LLM judges signals reliability

A Hacker News discussion examines the practice of using one large language model to grade another's output, and asks whether consensus among several such judges actually indicates a correct verdict. Commenters raise concerns that models can share the same blind spots or biases, so agreement may reflect correlated error rather than genuine quality. The thread touches on how evaluation setups should be validated, for example against human raters or adversarial examples.

evaluation/llm-as-judge/benchmarks/reliability