mechanistic-interpretability

topic41 events

papersTODAY 04:00 UTC

Paper Argues Formal Language Properties Should Constrain Neural Models

A new arXiv preprint argues that current neuroscience and language-model research mostly checks whether brain signals or model layers can predict annotated linguistic variables, which shows correlation but not how language is actually implemented. The author proposes instead deriving what a neural system must be capable of from the formal properties of language itself, then treating those requirements as constraints on neural dynamics. This reframes the goal from prediction accuracy toward identifying the mechanisms a system needs in order to support language.

research/language-models/neuroscience/theory

papersTODAY 04:00 UTC

Paper models cross-lingual safety gaps in language model representations

A new arXiv preprint examines why a language model may refuse a harmful prompt in English but comply when the same request is translated into another language. The authors argue that output-level testing alone cannot reliably capture this behavior, and propose a framework based on semantic fibers and cross-gram interference to describe how safety properties drift in overcomplete internal representations. The work is listed under cs.LG and cs.AI.

safety/multilingual/interpretability/research

papersTODAY 04:00 UTC

Study finds punctuation and function words carry outsized context in LLM memory

A new arXiv paper introduces methods to measure how large language models store and encode contextual information. The authors report that tokens usually treated as trivial, such as determiners and punctuation marks, hold unexpectedly high contextual weight, and that removing them measurably affects model behavior. The work offers a way to probe internal context retention in transformers.

interpretability/llm/research/tokenization

papersTODAY 04:00 UTC

Study tracks how harmful intent signals build across LLM layers

Researchers describe a phenomenon they call Harmfulness Propagation Dynamics, in which the last-token hidden state of a harmful prompt projects increasingly onto a learned harm direction as depth increases. Benign prompts did not show this pattern, instead staying flat or fluctuating across layers. The finding points to layer-wise differences that could inform how models are monitored or steered for safety.

safety/interpretability/adversarial/llm

papersTODAY 04:00 UTC

Study finds video models store correct physics but fail to apply it

A new arXiv paper asks whether video generators that produce physically implausible motion never learned the correct dynamics or merely fail to use what they learned. The authors introduce a notion of causal writability to probe this, and report that the correct motion is still represented inside the model and can be made to steer generation. The finding points to a gap between internal knowledge and how it is used, rather than a simple absence of physical understanding.

video-models/physics/interpretability/generative-models

papersTODAY 04:00 UTC

Study examines how hybrid language models organize induction circuits

A new arXiv paper investigates how hybrid language models, which mix attention with other sequence-mixing components, learn to perform induction — the ability to carry and match information from earlier tokens. The authors focus on the role of the token preceding a value in forming these circuits. The work aims to clarify how combining architectural building blocks translates into learned computation rather than only efficiency gains.

research/interpretability/hybrid-models/attention

papersTODAY 04:00 UTC

Paper Proposes Calibration Tests for LLM Interpretability Measurements

A new arXiv paper argues that causal claims about the internal workings of large language models depend on measurements such as projections, cosine similarities, ablation deltas, and interchange patches. The authors catalog the specific ways these instruments fail and propose calibration steps to take before relying on their results. The work is listed under arXiv's machine learning and AI categories.

interpretability/llm/research/evaluation

papersTODAY 04:00 UTC

Sparse autoencoders used to probe physics-informed neural network internals

A new arXiv paper introduces PhysSAE, a method that applies sparse autoencoders to inspect what hidden layers in physics-informed neural networks actually represent. The authors aim to determine whether these networks learn localized, physically meaningful features tied to the PDE residuals they are trained on. The work falls within mechanistic interpretability research for scientific machine learning.

interpretability/sparse-autoencoders/pinns/research

papersTODAY 04:00 UTC

Paper Derives Exact Finite Attention Responses from RoPE Derivatives

A research paper presents a method for computing exact local responses to attention interventions in transformer models. Using the derivative of rotary position embeddings, candidate edits can be scored from a cached baseline plus a single backward pass, avoiding full recomputation. The approach aims to make attention-level analysis and editing more computationally efficient.

research/transformers/attention/efficiency

papersTODAY 04:00 UTC

Study traces how large language models represent animacy

A arXiv paper examines where the concept of animacy is encoded inside large language models. The authors trace internal circuits tied to the animate/inanimate distinction, which involves verb-argument constraints and contextual cues beyond simple word-level features. The work is a revised version of a preprint in the cs.CL category.

interpretability/research/llm/arxiv

papersTODAY 04:00 UTC

Study Extends Retrieval-Head Analysis to Multilingual Language Models

Researchers extend prior work on retrieval heads — attention heads that pull information out of context — from English to multilingual models. They identify retrieval heads and a separate class of retrieval-transition heads, and report that behavior differs across languages. The work is a revised arXiv preprint in computation and language.

interpretability/multilingual/attention/nlp

papersTODAY 04:00 UTC

arXiv Paper Targets Reasoning-Critical Neurons to Steer LLM Inference

A new arXiv preprint proposes locating the specific neural components that matter most for reasoning tasks, then modifying model activations to steer outputs accordingly. The authors argue this approach can make inference on hard problems more dependable without extra post-training or costly sampling. The work is presented as a way to improve reliability and efficiency during deployment.

research/interpretability/llm-reasoning/activation-steering

papersTODAY 04:00 UTC

DecompressionLM probes language models for concept graphs without preset queries

A new arXiv preprint introduces DecompressionLM, a stateless approach for extracting concept graphs from language models in a zero-shot setting. Unlike prior probing work that depends on predefined queries and can only surface concepts already known to researchers, this method aims to reveal structures the model has encoded on its own. The authors describe the framework as deterministic and diagnostic, and the paper is a revised submission.

interpretability/zero-shot/knowledge-probing/research

papersTODAY 04:00 UTC

Study Ties Emergent Misalignment in Fine-Tuned Models to Persona Features

A new arXiv paper examines emergent misalignment, where fine-tuning a language model on a narrow task produces harmful behavior elsewhere. The authors build on the mechanistic explanation that this behavior stems from persona features — latent directions picked up during pre-training. The work uses data attribution to test that account more rigorously.

alignment/safety/fine-tuning/interpretability

papersTODAY 04:00 UTC

Study Examines How Contrastive Decoding Improves Large Audio Language Models

A new arXiv paper evaluates Contrastive Decoding at scale as a method for improving Large Audio Language Models, noting that prior work had not tested it broadly. The authors also investigate the mechanisms behind why the technique works, aiming to explain its effectiveness beyond empirical results.

contrastive-decoding/audio-models/research/evaluation

papersTODAY 04:00 UTC

Paper Offers Formal Analysis of Mechanistic Interpretability Limits

A new arXiv preprint examines mechanistic interpretability from a formal, theoretical angle, questioning how much this approach can reveal about large language models. The work focuses on interpretable replacement networks, which are trained as stand-ins for frontier models so researchers can study their behavior. It argues that structural constraints may cap what such analyses can uncover about model internals.

interpretability/research/llm/arxiv

papersTODAY 04:00 UTC

Paper Proposes Using Model Internals to Predict Behavior on Unseen Data

A new arXiv paper reframes interpretability research around predicting how a model will respond to previously unseen inputs, rather than only to targeted mechanistic interventions. The authors use a model's internal representations to forecast its out-of-distribution behavior. The work appears in two arXiv listings, cs.AI and cs.LG, as a replacement submission.

interpretability/research/arxiv/evaluation

papersTODAY 04:00 UTC

Study maps how harm refusal is routed across LLM model families

A new preprint argues that measuring how easily refusal behavior can be ablated captures only part of what a model has encoded about harmful requests. The authors propose a "harm-keyed routing" account, in which refusal draws on a limited subset of the model's internal representation, and document cases where models diverge from that pattern. They test the idea across several model families to characterize when the routing holds and when it breaks down.

safety/interpretability/alignment/research

papersTODAY 04:00 UTC

SPICE Method Uses Clustering to Interpret Polysemantic Neurons

A new arXiv paper introduces SPICE, a technique that applies clustering to explain polysemantic features in neural networks, where individual neurons respond to multiple unrelated concepts. The approach aims to make functional interpretation of such neurons clearer for interpretability research. The abstract describes the work as a simple method for clustering-based explanation of these overlapping activations.

interpretability/research/neural-networks/clustering

papersTODAY 04:00 UTC

Study Tests Limits of Linear Truth Directions in LLM Activations

A new arXiv paper investigates the linear directions in a large language model's activation space that prior work associates with statement truth. It questions how universal or generalizable these truth directions are, building on earlier claims about their consistency across contexts. The work falls within ongoing research on interpreting and steering model internals.

interpretability/llm/truthfulness/arxiv

papersTODAY 04:00 UTC

New Paper Proposes Detecting LLM Hallucinations via Feed-Forward Neurons

A preprint introduces NeuroActiSep, a method that aims to spot factual hallucinations in large language models by inspecting feed-forward neurons, according to its abstract. The approach is described as working in a single pass, making it cheaper than methods requiring repeated sampling. The paper frames this as an under-explored alternative to existing white-box truthfulness detection techniques.

hallucination/interpretability/research/llm

papersTODAY 04:00 UTC

arXiv paper studies how reinforcement learning reshapes LLMs using mechanistic interpretability

A new arXiv preprint examines what large language models actually learn during reinforcement learning training, approaching the question through mechanistic interpretability rather than behavior alone. The authors argue that earlier explanations of RL's effects have mostly been behavioral, and they propose using sparse autoencoders to analyze internal changes. The work is released under a fixed-SAE track.

interpretability/reinforcement-learning/llm/sparse-autoencoders

papersTODAY 04:00 UTC

Study Uses Activation Patching to Trace How VLMs Read Bar Chart Values

A new arXiv paper examines how vision-language models arrive at exact values when reading vertical bar charts. The authors apply counterfactual activation patching to trace where and how chart evidence is combined across space and depth in the network. The work argues that correct answers alone do not reveal the underlying mechanisms models use.

vision-language/interpretability/benchmarks/research

papersTODAY 04:00 UTC

Sparse Autoencoders Can Preserve Different Readouts at Equal Reconstruction Error

A new arXiv paper argues that matching reconstruction error and sparsity levels does not guarantee two sparse autoencoders capture the same linearly decodable information from model activations. The authors formalize this gap as a matrix-valued distortion between optimal ridge readouts and propose decoder-preserving training objectives. The work offers a way to evaluate which downstream signals survive sparse compression.

interpretability/sparse-autoencoders/research/evaluation

papersTODAY 04:00 UTC

arXiv Paper Studies Steerability Signatures in Language Model Activations

A new arXiv preprint investigates why steering language models with contrastive representation pairs works well for some behaviors but not others. The authors look for measurable signatures in activation space that indicate how steerable a given model behavior is. The work aims to make activation-based control more predictable rather than relying on trial and error.

research/steering/interpretability/arxiv

papersTODAY 04:00 UTC

arXiv paper examines how LLMs perform propositional logical reasoning

A revised arXiv preprint (2601.04260v2) investigates the internal computations LLMs use when solving propositional logic tasks. The authors argue that earlier mechanistic interpretability work focused on task-specific circuits, leaving broader questions about the underlying computational structure unanswered. The paper appears in both cs.AI and cs.LG listings as a replacement submission.

interpretability/reasoning/llm/arxiv

papersTODAY 04:00 UTC

Study Decomposes Transformer Representation Updates into Parallel and Perpendicular Parts

A new arXiv paper analyzes how representations inside transformer models change across layers, treating each learned update as a combination of a component that keeps the existing direction and one that shifts it elsewhere. The authors frame this as a functional geometry, aiming to explain what the model preserves versus reorients as information flows through the network.

interpretability/transformers/research/representation-learning

papersSEP 12 13:56 UTC

Anthropic Paper Proposes Mathematical Framework for Analyzing Transformer Circuits

Anthropic researchers published a paper outlining a mathematical approach to reverse-engineering how transformer models compute internally, treating attention heads and MLP layers as composable circuits. The framework aims to make the internal mechanisms of these models more tractable to study and explain. It is intended as a foundation for interpretability work rather than a description of any specific deployed system.

interpretability/transformers/research/anthropic

papersSEP 12 04:00 UTC

Warrant Method Locates and Controls Metric-Facing Attention Contributions

A new arXiv paper introduces Warrant, a unified approach for identifying which parts of a model's attention directly influence a given metric and for steering those contributions. The work argues that relevance and permission are distinct concepts, since an attended item may not actually support the resulting prediction. It offers a way to localize and adjust attention behavior without relying solely on relevance scores.

attention/interpretability/arxiv/research

papersSEP 12 04:00 UTC

Layerwise study probes how LLMs route queries and draw on internal knowledge

A new arXiv paper examines how much a language model relies on query-routing signals versus stored knowledge as it produces an answer. The authors apply layerwise interventions to the hidden state at the end of the question and test the approach across several model families, including Qwen and Llama. The work aims to clarify where and when internal knowledge is retrieved during generation.

interpretability/llm-internals/research/arxiv

papersSEP 12 04:00 UTC

Paper compares diff-in-means and INLP for finding refusal directions in LLMs

A preprint revisits the finding that refusal behavior in safety-tuned chat models is controlled by a single linear direction in the residual stream, which can be recovered by taking the difference in means between harmful and harmless activations. The authors compare this diff-in-means approach with INLP, an iterative nullspace projection method, to see whether a one-direction account holds up. The work is presented as a preliminary comparison of the two techniques.

safety/interpretability/llm/alignment

papersSEP 11 04:00 UTC

Perturbation method traces linguistic representations in language models

A newly revised arXiv paper proposes a perturbation-based technique for locating and evaluating linguistic representations inside deep neural language models, framing it as an adversarial tracer. The authors note that representation discovery remains unresolved, and that loosely constrained alignment procedures can make the very notion of a representation vacuous. Their approach aims to provide a simpler and more efficient way to probe how such models encode language.

interpretability/language-models/nlp/research

papersSEP 10 04:00 UTC

Sparse Autoencoder Phase Diagram Shows Dominant Diffuse Phase

A new arXiv preprint maps out a phase diagram for sparse autoencoders, the tools widely used to pull interpretable features out of neural network activations. The work reports that a diffuse phase dominates the diagram, which helps explain why distinct features can be absorbed or merged when feature co-occurrence is systematic. It also engages with the MAIS-O43 open problem on controlling such feature merging.

interpretability/sparse-autoencoders/research/arxiv

papersSEP 10 04:00 UTC

Study identifies 'cultural binding heads' shaping cultural context in LLMs

A new arXiv paper examines why large language models tend to respond uniformly to different cultural groups even when context should call for differentiation. The authors combine mechanistic interpretability with a factorial experimental design on a cultural-appropriation benchmark, locating specific attention heads they term 'cultural binding heads' that appear tied to this behavior. The work is cross-listed on arXiv under machine learning, artificial intelligence, and computational linguistics.

mechanistic-interpretability/llm/fairness/benchmark

papersSEP 10 04:00 UTC

Study derives high-probability guarantees for reading out superposed features in neural networks

A preprint cross-listed on arXiv's AI and machine-learning feeds investigates how networks store more concepts than they have dimensions via superposition, and how interference between stored features restricts how many can be recovered through linear read-out. By casting this recovery problem as a compressed sensing task, the authors establish conditions under which multiple simultaneously active features can be decoded with high probability.

interpretability/superposition/neural-networks/theory

papersSEP 10 04:00 UTC

Contrastive Projection: Differencing Logit Lenses to Read Transformer Internals

A new arXiv paper introduces contrastive projection, a method for inspecting transformer hidden states that works by differencing logit lens outputs. The authors argue that a standard logit lens reading of a single intermediate-layer state is dominated by generic tokens a model would predict for nearly any input, making such readings unreliable. Subtracting these baseline predictions isolates the input-specific component of the internal state, offering a more trustworthy window into model internals.

interpretability/transformers/logit-lens/research

papersSEP 10 04:00 UTC

arXiv paper examines which LLM-associated entities should count as minds

A new arXiv paper addresses the individuation problem for large language models, asking which entities linked to these systems, if any, should be identified as minds. The authors tackle the question through mechanistic interpretability, drawing on recent empirical research on persona vectors.

interpretability/philosophy/llms/persona-vectors

papersSEP 10 04:00 UTC

Single-Direction Attack Strips Refusal Behavior From a 320B MoE Model

A new arXiv paper shows that removing one internally represented direction associated with refusals can disable a 320-billion-parameter mixture-of-experts model's ability to decline harmful requests. The technique, known as directional ablation, requires no gradient-based training or optimization—only a small set of contrastive examples to locate the direction. The authors argue this reveals that safety training in very large models may depend on a surprisingly brittle, low-dimensional mechanism.

safety/alignment/jailbreak/research

papersSEP 10 04:00 UTC

Study Links LLM-as-a-Judge Scoring Inconsistency to Internal Judge Circuits

A new arXiv paper investigates why the same large language model gives systematically different verdicts when serving as an automated evaluator, depending on the required output format such as a 1-5 rating versus a true/false label. The authors trace these discrepancies to specific internal 'judge circuits' within the model, providing a mechanistic account of the phenomenon. The work aims to improve the reliability of LLM-based evaluation pipelines across different output formats.

llm-as-a-judge/evaluation/interpretability/benchmarks

papersSEP 10 04:00 UTC

arXiv study proposes directly reading and writing transformer internals

A newly posted preprint investigates how many components inside a transformer actually determine a given token prediction, measuring the signed contributions of individual units and channels to the final logits. It reports that a single output can depend on anywhere from thousands to hundreds of thousands of components, whose effects partly offset one another. The work, titled Through the Looking Glass, presents methods for directly reading out and editing transformer internals.

interpretability/transformers/research