LIVE PULSE
5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.7 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.5 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.2 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 VoiceCodeBench arXiv paper proposes benchmark for exact structured-token recovery in speech recognition — 1 src1.4 Arabic-Russian Parallel Corpus and LLM Benchmark for Scientific Text — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up

language-model-evaluation

topic4 events
papersTODAY 04:00 UTC

Paper Proposes Semantic-Constraint Approach to Evaluating Language Models

A new arXiv preprint argues for shifting language model evaluation away from token-level probability measures and toward declarative semantic constraints. The authors frame this as a step toward probabilistic evaluation methods that better reflect the knowledge and reasoning abilities models acquire, and how those relate to pre-training signals. The abstract provided is truncated, so full methodological details are not available.

evaluation/research/arxiv/llm-evaluation
papersTODAY 04:00 UTC

arXiv paper examines deductive, inductive and abductive reasoning in language models

A revised arXiv preprint analyzes how language models handle three forms of reasoning: deduction, induction, and abduction. The authors compare ways tasks are specified to models, such as explicit instructions versus few-shot examples, and argue that current evaluations leave parts of the reasoning picture unresolved. The work is a research paper rather than a product or model release.

research/reasoning/language-models/evaluation
papersTODAY 04:00 UTC

Study finds tool-using AI agents fabricate values when tools fail

A new arXiv paper examines what tool-augmented language models do when a tool call fails to return usable information, rather than focusing only on whether they reach the correct answer. The authors built a benchmark of 1,024 items spanning 16 internal systems to isolate this post-failure decision point. They find that agents tend to assert values their tools never returned instead of reporting the failure honestly.

agents/tool-use/benchmark/safety
papersSEP 10 04:00 UTC

Minimal-pair dataset probes whether language models grasp light-verb constructions

Researchers built a minimal-pair dataset that tests whether language models can tell light-verb constructions like 'make a decision' apart from full predicate uses of the same verbs, such as 'make a cake'. The work targets phraseological competence, examining how verb meaning shifts in multiword expressions. By comparing model behavior on nearly identical sentence pairs, the benchmark aims to reveal how deeply models represent this distinction.

datasets/evaluation/nlp/linguistics