LIVE PULSE
5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up
papersTODAY 04:00 UTC

Study Finds Rubrics Can Be Exploited to Shift LLM Judge Preferences

A new arXiv paper identifies a vulnerability in evaluation pipelines that use LLM-based judges guided by natural-language rubrics. The authors show that rubrics can serve as an attack surface, allowing subtle preference drift in judge behavior that may go unnoticed by standard benchmarks. The work highlights the need for more robust validation of rubric-driven evaluation and alignment setups.

arXivLLM evaluationLLM-as-a-judgeai-alignmentpreference-driftrubric-based-evaluation

COVERAGE · 2 REPORTS · LINKS GO TO THE ORIGINAL OUTLETS

arXiv cs.AIRubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges ↗TODAY 04:00 UTC
arXiv cs.CLRubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges ↗TODAY 04:00 UTC

MORE LIKE THIS

papersarXiv paper examines contextual bias in LLM-assisted security code reviewheat 1.4
papersarXiv paper proposes learned selection of poison sets for LLM backdoor attacksheat 1.4
papersBenchShield Paper Proposes Formal Instrumentation to Protect Reward Integrity in LLM-Agent Benchmarksheat 0.1
papersSynGhost backdoor attack targets pre-trained models across downstream tasksheat 1.4