#instruction-following — HeatPulse — HeatPulse

LIVE PULSE

5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src

#instruction-following

2 curated events

papersTODAY 04:00 UTC

MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

A new arXiv paper introduces MTAC-IFBench, a benchmark aimed at measuring how well large language model agents follow instructions across multi-turn coding sessions. The work targets agentic software engineering, where models plan, run code, and call external tools over successive steps rather than producing a single answer. It addresses evaluation beyond functional correctness, focusing on whether agents keep to the constraints given to them.

agents/benchmark/code-generation/instruction-following

papersTODAY 04:00 UTC

IBBench-Light benchmark tests whether models treat external records as instructions or text

A new arXiv paper introduces IBBench-Light, an evaluation that presents the same external record to a model under two different uses: as a procedure the model must carry out, or as text it must simply read. Each of twelve semantic bases produces 144 matched response pairs per model, and four quantized instruction-tuned models were tested. The paired setup is meant to isolate whether models react to a directive's form or to the user's stated task.

benchmarks/evaluation/instruction-following/quantization