#attention

21 curated events

papersTODAY 04:00 UTC

Cross-Modal Attention Network Targets Speech Biomarkers of Cognitive Decline

A new arXiv paper introduces CCMAN, a cross-modal attention model designed to detect early cognitive decline from verbal fluency speech tasks. Unlike prior approaches that pool features over an entire recording, the method explicitly accounts for cognitive instability and aims to produce interpretable temporal biomarkers. The work is framed as a scalable, non-invasive complement to conventional clinical assessment.

speech/healthcare/interpretability/attention

papersTODAY 04:00 UTC

Coresets for Softmax Attention: Improved Bounds and Efficient Constructions

A new arXiv paper studies query-oblivious coresets for softmax attention heads, defined as subsets of key-value pairs that approximate the full attention output within an error bound for every query in a ball. The authors build on earlier work by Liberty, Andoni and Kleiner on unweighted coresets to derive tighter bounds and faster construction methods. The goal is to reduce the compute and memory needed when evaluating attention over long key-value sets.

attention/efficiency/coresets/research

papersTODAY 04:00 UTC

Study Extends Retrieval-Head Analysis to Multilingual Language Models

Researchers extend prior work on retrieval heads — attention heads that pull information out of context — from English to multilingual models. They identify retrieval heads and a separate class of retrieval-transition heads, and report that behavior differs across languages. The work is a revised arXiv preprint in computation and language.

interpretability/multilingual/attention/nlp

papersTODAY 04:00 UTC

GEAR: From Dynamic Encoding to Dynamic Activation in Social Trajectory Prediction

A new arXiv paper introduces GEAR, a method for predicting human trajectories that must account for both how individuals move and how agents influence one another. The authors note that prior work has relied on attention mechanisms, graph structures, and temporal encoders to model these dynamics, and propose shifting the emphasis from dynamic encoding to dynamic activation. The work appears as a cross-listing on arXiv's cs.AI category.

arxiv/trajectory-prediction/attention/research

papersTODAY 04:00 UTC

Study examines how hybrid language models organize induction circuits

A new arXiv paper investigates how hybrid language models, which mix attention with other sequence-mixing components, learn to perform induction — the ability to carry and match information from earlier tokens. The authors focus on the role of the token preceding a value in forming these circuits. The work aims to clarify how combining architectural building blocks translates into learned computation rather than only efficiency gains.

research/interpretability/hybrid-models/attention

papersTODAY 04:00 UTC

Attention-Based Flux Scheme Targets Spurious Oscillations in Finite-Volume Solvers

A new arXiv preprint proposes a conservative finite-volume method on fixed grids in which an attention mechanism selects upstream information to build the numerical flux. The scheme is conditioned on the CFL constraint so that a shock can propagate across multiple cells in a single update without being smeared or broken. The work sits at the intersection of numerical PDE solvers and machine learning attention architectures.

attention/numerical-methods/simulation/arxiv

papersTODAY 04:00 UTC

Paper Derives Exact Finite Attention Responses from RoPE Derivatives

A research paper presents a method for computing exact local responses to attention interventions in transformer models. Using the derivative of rotary position embeddings, candidate edits can be scored from a cached baseline plus a single backward pass, avoiding full recomputation. The approach aims to make attention-level analysis and editing more computationally efficient.

research/transformers/attention/efficiency

papersTODAY 04:00 UTC

Audio language models track speakers via text backbone attention, study finds

A new study examines how audio language models attribute speech to the correct speaker, finding accuracy of only 6 to 16 percent on a six-speaker task, below random guessing. The authors show that speaker tracking relies on attention heads in the model's text backbone, and that altering a subset of those heads shifts which speaker the model retrieves.

audio/attention/multimodal/research

papersTODAY 04:00 UTC

Biquaternionic Space with Complex-Valued Attention for Temporal Knowledge Graph Completion

A new arXiv preprint proposes embedding temporal knowledge graphs in biquaternionic space, arguing that relying on a single geometric space limits how well models capture varied relational patterns. The method pairs that representation with complex-valued attention to score facts whose validity changes over time. The work targets the link-prediction task of inferring missing facts in evolving knowledge graphs.

knowledge-graphs/embeddings/attention/research

papersTODAY 04:00 UTC

Paper proposes transport criterion for exact record omission in delta attention

A new arXiv paper examines the gap between a model appearing to forget a deleted record and its recurrent state still retaining that record's influence. The authors store the state difference produced right after a record is written and track it through subsequent processing steps, deriving a condition under which the omission is exact along with its computational cost and a replay-based certificate. The work targets attention mechanisms with recurrent memory, where deleting data from context does not necessarily remove its effect from internal state.

research/attention/memory/unlearning

papersTODAY 04:00 UTC

Paper proposes content-based addressing to replace positional rotation in long-context attention

A new arXiv preprint argues that rotary position embeddings (RoPE) degrade as context windows grow, because the integer positions seen during inference differ from those used in training. The authors suggest determining attention rotation from content-based addressing rather than from a token's fixed index, aiming to remove this train-test mismatch. The work is a methodological proposal and cross-listing revision, not a released model or product.

long-context/position-embeddings/arxiv/attention

papersTODAY 04:00 UTC

Paper Proposes Attention-Based Method for Multivariate Time Series Anomaly Detection

A revised arXiv paper introduces a technique that flags anomalies in multivariate time series by tracking shifts in cross-channel dependencies rather than only large amplitude changes. The authors illustrate the idea with autonomous driving, where a steering command can look internally consistent yet no longer match the resulting vehicle behavior. The work appears on arXiv under cs.AI and cs.LG as a cross-listing update.

anomaly-detection/time-series/attention/arxiv

papersTODAY 04:00 UTC

AttnFuse: A Composable DSL for Compiling Attention Variants into Fused GPU Kernels

Researchers present AttnFuse, a domain-specific language that compiles attention implementations into fused GPU kernels. The work targets the fact that attention dominates compute and memory costs in Transformer models, while new attention variants keep proliferating. The DSL aims to let developers describe such variants and get fused kernels without hand-writing low-level GPU code.

attention/gpu-kernels/dsl/transformers

papersTODAY 04:00 UTC

Grouped Value Attention Cuts KV Cache Cost via On-Demand Key Reconstruction

A new arXiv paper proposes Grouped Value Attention, a method that lowers the memory and read-bandwidth burden of the key-value cache during Transformer decoding. Unlike grouped-query attention, which still stores separate keys and values per shared head, the approach reconstructs keys on demand from stored values. This targets the KV cache, which the authors identify as a main bottleneck as sequence lengths grow.

kv-cache/efficiency/attention/inference

papersSEP 10 04:00 UTC

Predicting Middle-Layer Attention in Multimodal LLMs for Efficient Visual Token Pruning

Multimodal large language models spend significant compute processing large numbers of visual tokens, and effective pruning depends on knowing which tokens actually matter. This paper introduces a learned approach that predicts attention at middle layers, enabling models to identify and drop less relevant visual tokens. The method aims to cut inference costs while maintaining performance across vision-language tasks.

multimodal/efficiency/token-pruning/attention

papersSEP 10 04:00 UTC

Time-Frequency Geometric Cross-Attention for Chunked Vision-Language-Action Models

A new arXiv paper proposes a time-frequency geometric cross-attention mechanism for vision-language-action policies that emit chunks of actions in one forward pass. The authors argue that an action chunk is effectively a short multivariate trajectory and design their architecture to model it as such. The work targets robotic control models that generate one to two seconds of coordinated motion per prediction.

robotics/attention/vla

papersSEP 10 04:00 UTC

Spectral framework examines phase structure in rotary attention of Transformers

A cs.CL preprint argues that rotary position encoding gives Transformer attention a phase structure that typical vector-geometry analyses overlook. The authors develop a bounded spectral framework to study how queries and keys interact under this phase-based view. They apply it to semantic continuity and to governing where model execution begins and ends.

transformers/attention/spectral-analysis/interpretability

papersSEP 10 04:00 UTC

EFQ-Softmax Method Aims to Remove Exponentials from Attention Quantization

A new arXiv paper proposes EFQ-Softmax, a technique for quantizing the softmax step in Transformer attention without computing exponentials. Low-bit attention already speeds up inference by shifting matrix multiplications to FP8 or FP4 engines, but the softmax path often still needs higher precision for shifted-score exponentials and temporary probability storage. The work targets that remaining bottleneck in the attention pipeline.

quantization/transformers/attention/inference

papersSEP 10 04:00 UTC

Cross-Attention Model Improves Cardiovascular Event Prediction from Medical Claims

A new arXiv paper describes a cross-attention approach for predicting major adverse cardiovascular events using medical claims data, which combines billing and clinical records. The method aims to improve on existing predictive models by better integrating these two types of information. The work is a preprint and has not yet been peer reviewed.

healthcare/research/attention/claims-data

papersSEP 10 04:00 UTC

New Relation mechanism decouples relation formation from flow allocation in token mixing

A revised arXiv paper in machine learning introduces Relation, a token-mixing mechanism that splits an operation standard attention fuses into a single score-to-flow step. The method first organizes pairwise evidence into explicit Self and Exchange relations and then allocates information flow across them. The authors position this as an alternative to dominant attention-based token mixing in sequence models.

attention/architecture/deep-learning/transformers

papersSEP 12 04:00 UTC

Warrant Method Locates and Controls Metric-Facing Attention Contributions

A new arXiv paper introduces Warrant, a unified approach for identifying which parts of a model's attention directly influence a given metric and for steering those contributions. The work argues that relevance and permission are distinct concepts, since an attended item may not actually support the resulting prediction. It offers a way to localize and adjust attention behavior without relying solely on relevance scores.

attention/interpretability/arxiv/research