LIVE PULSE
5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src5.1 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.8 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.6 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src2.3 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.8 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.4 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.4 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.4 Fine-Tuning Vision-Language Models with Listener Gaze for Referring Expressions — 1 src1.4 Perceptual Reality Transformer Explores What Illustrations Must Preserve — 1 src1.4 Study Analyzes Self-Reported Limitations in NLP Research — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up

AI auditing

topic11 events
papersTODAY 04:00 UTC

Paper Argues Pair Counts Overstate Coverage in Transformation Audits

A new arXiv paper contends that reporting the number of equivalent pairs is a misleading way to describe how much an audit actually constrains a model. Because pairs derived from the same underlying object are correlated, and full orbit graphs repeat much of the same information, raw pair totals can make an evaluation look more thorough than it is. The authors recommend measuring coverage in terms of the distinct constraints an audit imposes rather than simple pair counts.

evaluation/robustness/research/benchmarks
papersTODAY 04:00 UTC

Image-swap audit tests whether medical vision-language models rely on scans

A new arXiv paper introduces ModaLens, an audit method that swaps paired images to check how much a medical vision-language model actually depends on the image versus the accompanying radiology report. Because a report may already contain the answer to a clinical question, the technique measures how image sensitivity shifts when report text is available. The authors report results for MedGemma-27B among other models.

medical-ai/vision-language/evaluation/arxiv
papersTODAY 04:00 UTC

CALICO System Aligns LLM Annotation Prompts With Expert Codebooks

Researchers present CALICO, a human-centered system that helps domain experts turn their annotation codebooks into prompts for large language models. The work targets gaps in existing pipelines, which offer little support for producing prompts that stay reliable, easy to revise, and auditable. It is described in a paper posted to arXiv under the cs.CL category.

annotation/llm/research/human-in-the-loop
papersTODAY 04:00 UTC

Paper Proposes Auditable Method for Verifying Deletion of Stored Facts in AI Models

The paper installs a support-vector gate inside a frozen Gemma 3 model and records which stored keys and values correspond to each exchange, allowing a deletion request to be traced to specific memory entries. It argues that an assistant can stop repeating a fact without actually removing it, so the work separates genuine deletion from mere output suppression. The authors present this as a step toward deletion requests that can be audited rather than taken on trust.

machine-unlearning/safety/memory/gemma
papersTODAY 04:00 UTC

DepthBenchCAD Examines Whether More Auditing Checks Improve Generative CAD Evaluations

A new arXiv preprint introduces DepthBenchCAD, a benchmark studying how the number of edit checks affects the reliability of evaluations for generative CAD models. The work focuses on behavioral correctness after parameter edits and asks whether auditing more programs under a fixed budget actually leads to firmer conclusions. It questions the common assumption that adding edit checks is a straightforward path to more trustworthy evaluation.

benchmark/cad/evaluation/generative-models
papersTODAY 04:00 UTC

Frozen Physiological Encoder Keeps ICU Model Explanations Stable During Updates

A new arXiv paper proposes updating intensive care prediction models through a structurally bounded procedure that leaves the physiological encoder frozen. The authors argue this limits how much model behavior and its explanations can drift when patient data distributions change. The aim is to make adapted clinical models easier to audit after deployment.

healthcare/interpretability/model-updates/arxiv
papersSEP 12 15:15 UTC

Chain-of-thought reasoning: from Google research to closing AI transparency

Chain-of-thought prompting, formalized by Google researchers in 2022, pushed models to work through problems step by step and changed how AI systems tackle reasoning tasks. The technique is now built natively into OpenAI's models, but the visibility it once offered into a model's internal steps is narrowing. That shift raises concerns about how developers and regulators can audit increasingly capable systems.

chain-of-thought/interpretability/safety/openai
papersSEP 12 04:00 UTC

Survey Maps Evidence Tracing and Provenance Methods for LLM Agents

A new arXiv survey examines how evidence tracing and execution provenance can be applied to LLM-based agents that plan, call tools, retrieve information, and collaborate across multiple agents. The authors frame provenance tracking as a way to make agent behavior auditable and to build trust as these systems take on more autonomous tasks. The paper is a revision of an earlier preprint.

agents/research/trust/auditability
papersSEP 10 04:00 UTC

Study finds LLMs degrade as error auditors with batch size, hallucinating confidently

Researchers assembled a corpus of 150 academic papers with deliberately planted errors to test how well large language models can act as automated document-quality auditors. They report that detection reliability worsens as processing batch sizes increase, and that models sometimes fabricate audit findings with high confidence. The results cast doubt on deploying LLMs unsupervised for contamination-detection tasks.

hallucination/evaluation/reliability/auditing
papersSEP 10 04:00 UTC

AgentAudit: Open Framework Evaluates AI Agents Across Their Full Lifecycle

A new arXiv paper introduces AgentAudit, an open and extensible framework for auditing the trustworthiness of AI agents. The authors contend that today's benchmarks examine only slices of agent behavior, such as task success or robustness against attacks, and instead propose measuring every stage of an agent's operation, including planning, tool use, memory, and reasoning. The design is meant to be extendable so that new evaluation checks can be added over time.

agents/evaluation/open-source/safety
papersSEP 10 04:00 UTC

Researchers Propose Discovery Certification Protocol for Auditing AI Research Agents

An arXiv paper argues that benchmark scores by themselves are not sufficient evidence that AI research agents have made genuine scientific discoveries. The authors introduce the Discovery Certification Protocol, which converts an agent's claimed results into executable recovery and feedback tests that can be independently run. The first of its gates checks that reported results can actually be reproduced from the recorded evidence.

agents/evaluation/reproducibility/research