S-CEReBrO Architecture Targets Memory Limits in Continuous EEG Monitoring

A new arXiv paper introduces S-CEReBrO, an approach aimed at overcoming the memory constraints that arise when applying Transformer-based foundation models to long-running EEG recordings. Global attention scales poorly with signal length, which limits how far such models can be used for continuous brain-monitoring data. The work proposes a redesign intended to make long-sequence EEG analysis more tractable while retaining the generalization benefits of large pretrained models.

T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning

A new arXiv preprint introduces T-LoopFormer, a looped Transformer variant that varies how many times each token passes through a shared parameter block rather than applying a fixed depth uniformly. The approach uses dynamic routing to decide token-level elastic depth, aiming to support latent reasoning while keeping parameter counts low. The announcement is an abstract-only submission, so reported results and comparisons are not yet detailed.

SpecAugment-Patch Merging Proposed to Speed Up Audio Spectrogram Transformer Training

A new arXiv paper introduces SpecAugment-Patch Merging, a method that masks input spectrograms at the patch level before positional embeddings are added, then merges patches to cut computation. The authors describe it as a simple approach to accelerate training of Audio Spectrogram Transformers. The work is categorized under machine learning research.

Paper Proposes Affinity-Aware Sharding to Improve Delayed Tensor Parallelism

A new arXiv preprint examines Delayed Tensor Parallelism (DTP), a scheme that avoids the blocking all-reduce step in tensor-parallel Transformer inference by letting each device immediately add its partial output to the residual stream and broadcast it. The authors introduce affinity-aware sharding as a way to organize this communication and computation across devices. The work targets more efficient distributed inference for large Transformer models.

Paper compares looped and stacked transformer encoders on 12-lead ECG

A new arXiv preprint examines how recurrent transformers that reuse a single set of weights differ from conventional stacked transformers in representation and dynamics. Using 12-lead electrocardiogram data as the test domain, the authors analyze the mechanistic differences between the two designs. The work aims to clarify the trade-offs behind the parameter efficiency that has made looped architectures popular.

Paper Derives Exact Finite Attention Responses from RoPE Derivatives

A research paper presents a method for computing exact local responses to attention interventions in transformer models. Using the derivative of rotary position embeddings, candidate edits can be scored from a cached baseline plus a single backward pass, avoiding full recomputation. The approach aims to make attention-level analysis and editing more computationally efficient.

Study Decomposes Transformer Representation Updates into Parallel and Perpendicular Parts

A new arXiv paper analyzes how representations inside transformer models change across layers, treating each learned update as a combination of a component that keeps the existing direction and one that shifts it elsewhere. The authors frame this as a functional geometry, aiming to explain what the model preserves versus reorients as information flows through the network.

Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

A new arXiv paper proposes a transformer-based meta-controller for adaptive control of robotic manipulators that must cope with friction memory they cannot directly observe. Existing attention-based controllers fix the number of attention heads before training and rely on expensive offline tuning; this work instead adjusts capacity incrementally at runtime. The authors position the method as a reinforcement-learning approach to adaptive control in a cross-listed machine learning submission.

Transformer Model and Mandarin Speech Dataset Target Audio-Based Kinship Verification

Researchers introduce CONVTRAP-TN, a transformer-based architecture designed to determine whether two speakers share a first-order family relationship using only audio. The work also releases a new uncontrolled Mandarin kinship speech dataset and reports an ablation study on the model's components. Kinship verification from voice is a relatively underexplored area, and the Mandarin dataset addresses a gap in multilingual resources.

Transformer Model Detects Schizophrenia from EEG Spectrograms

A new arXiv paper presents a transformer-based framework that analyzes EEG signals converted into spectrogram form to identify schizophrenia. The approach aims to support diagnosis, which currently relies mainly on clinical evaluation, by using a non-invasive brain-activity measurement. The work is a preprint and has not yet been validated in clinical settings.

Study compares eight tokenization strategies for ECG transformer models

A new arXiv paper examines how different tokenization choices affect ECG transformer models, since the tokenizer decides both the physiological signal content the model sees and the sequence length attention operates over. The authors benchmark eight tokenization strategies across four architectures — Transformer, Informer, Reformer, and FEDformer — on the nine-label CPSC ECG dataset. The work is cross-listed in cs.AI and cs.LG.

AttnFuse: A Composable DSL for Compiling Attention Variants into Fused GPU Kernels

Researchers present AttnFuse, a domain-specific language that compiles attention implementations into fused GPU kernels. The work targets the fact that attention dominates compute and memory costs in Transformer models, while new attention variants keep proliferating. The DSL aims to let developers describe such variants and get fused kernels without hand-writing low-level GPU code.

Attention Bridge Method Distills Transformers into Mamba Models with Less Data

A new arXiv paper proposes an "attention bridge" technique for converting pretrained Transformer models into Mamba-style state-space models. The approach aims to make the distillation process more data efficient, addressing the high compute cost of training competitive SSMs from scratch. The work targets the gap between the mature Transformer ecosystem and the less developed tooling around state-space architectures.

arXiv Paper Proposes Synchronization-Based Attention for Low-Energy Hardware

A new arXiv preprint explores replacing standard softmax attention with a mechanism based on synchronization in networks of coupled oscillators. The authors argue that exponentiation and global reduction are costly on conventional von Neumann hardware and lack a direct physical counterpart, unlike coupled-oscillator dynamics such as Kuramoto models. The work targets transformer-style attention on energy-constrained physical substrates.

Anthropic Paper Proposes Mathematical Framework for Analyzing Transformer Circuits

Anthropic researchers published a paper outlining a mathematical approach to reverse-engineering how transformer models compute internally, treating attention heads and MLP layers as composable circuits. The framework aims to make the internal mechanisms of these models more tractable to study and explain. It is intended as a foundation for interpretability work rather than a description of any specific deployed system.

Spectral framework examines phase structure in rotary attention of Transformers

A cs.CL preprint argues that rotary position encoding gives Transformer attention a phase structure that typical vector-geometry analyses overlook. The authors develop a bounded spectral framework to study how queries and keys interact under this phase-based view. They apply it to semantic continuity and to governing where model execution begins and ends.

Central Dogma Transformer II: an interpretable model for cellular gene regulation

A revised arXiv preprint introduces Central Dogma Transformer II, a transformer-based model framed as an 'AI microscope' for studying cellular regulatory mechanisms. The authors argue that gene-regulation research requires models whose learned internal structure can be directly examined and mapped onto regulatory relationships, rather than opaque but accurate predictors. The update refines the paper's case for interpretability as a core requirement in computational biology.

Predicting Electrical Outage Restoration Times with Longitudinal Tabular Transformers

Researchers have revised a study on predicting Estimated Times of Restoration, the timelines utilities publish for storm-related power outages. The work reframes ETR prediction as a longitudinal task, using tabular transformer models instead of the static approaches used in earlier research. Because restoration estimates shape customer decisions about food storage, medical devices, and evacuation, improving their accuracy has direct practical value.

Training trajectories determine circuit removability in annealable soft-prior Transformers

Researchers asked whether retrieval circuits that small Transformers learn with the help of soft positional priors keep functioning once that prior is taken away. They tested this using a model whose prior-based attention biases can be gradually annealed out during training. The results indicate that the specific training trajectory, not just the architecture, decides whether a learned circuit can stand on its own after the prior is removed.

New Relation mechanism decouples relation formation from flow allocation in token mixing

A revised arXiv paper in machine learning introduces Relation, a token-mixing mechanism that splits an operation standard attention fuses into a single score-to-flow step. The method first organizes pairwise evidence into explicit Self and Exchange relations and then allocates information flow across them. The authors position this as an alternative to dominant attention-based token mixing in sequence models.

Decision Transformer applied to UAV-mounted RIS-assisted dynamic D2D communications

A new arXiv paper proposes using a decision transformer to control a reconfigurable intelligent surface mounted on a UAV to support device-to-device links whose activation is stochastic. The approach models UAV motion and attitude along with time-varying Rician channel angles, adapting the surface configuration as conditions change.

Elastoformer paper proposes elastic model transformation for adaptive edge AI

A new arXiv paper introduces Elastoformer, a method that reshapes transformer models at runtime using elastic transformations so they can adjust to shifting operating conditions. The work targets computer vision workloads on edge devices, where latency, power, and compute budgets fluctuate. By adapting model capacity on the fly, the approach aims to let on-device systems balance accuracy against resource constraints in real time.

Contrastive Projection: Differencing Logit Lenses to Read Transformer Internals

A new arXiv paper introduces contrastive projection, a method for inspecting transformer hidden states that works by differencing logit lens outputs. The authors argue that a standard logit lens reading of a single intermediate-layer state is dominated by generic tokens a model would predict for nearly any input, making such readings unreliable. Subtracting these baseline predictions isolates the input-specific component of the internal state, offering a more trustworthy window into model internals.

Study Quantifies Logical Consistency in Transformers via Query-Key Alignment

A new arXiv paper proposes a technique for measuring how logically consistent transformer language models are during multi-step reasoning, using analysis of query-key alignment in attention mechanisms. The authors position this as a way to assess reasoning reliability beyond improvements from prompting methods such as Chain-of-Thought. The work appears on arXiv under cs.AI (2502.17017).

arXiv study proposes directly reading and writing transformer internals

A newly posted preprint investigates how many components inside a transformer actually determine a given token prediction, measuring the signed contributions of individual units and channels to the final logits. It reports that a single output can depend on anywhere from thousands to hundreds of thousands of components, whose effects partly offset one another. The work, titled Through the Looking Glass, presents methods for directly reading out and editing transformer internals.

EFQ-Softmax Method Aims to Remove Exponentials from Attention Quantization

A new arXiv paper proposes EFQ-Softmax, a technique for quantizing the softmax step in Transformer attention without computing exponentials. Low-bit attention already speeds up inference by shifting matrix multiplications to FP8 or FP4 engines, but the softmax path often still needs higher precision for shifted-score exponentials and temporary probability storage. The work targets that remaining bottleneck in the attention pipeline.

Study challenges the narrow-wide-narrow FFN convention in Transformer language models

An arXiv research paper questions why dense Transformers almost universally place most of their non-embedding parameters in narrow-wide-narrow feed-forward networks. Drawing on theoretical and empirical evidence, the authors explore an alternative wide-narrow-wide (hourglass) residual design for these blocks. The work is cross-listed across the cs.AI, cs.CL, and cs.LG categories on arXiv.

TempTPI: Informer-Based Trajectory Prediction for Maritime Vessels

A new arXiv paper introduces TempTPI, a model that adapts the Informer architecture to forecast the future paths of ships over long time horizons. The approach learns from AIS tracking data, where conventional Transformers can struggle with long input sequences. The authors target improvements in navigational safety and logistics planning for the maritime sector.

SG-Blend Activation Function Combines Swish and GELU for Neural Networks

Researchers propose SG-Blend, an activation function that interpolates between an improved version of Swish and GELU rather than relying on either alone. The work addresses the fact that Swish was found through architecture search on vision tasks while GELU became standard in transformer language models, leaving neither optimal across domains. The paper is a revised cross-listing on arXiv.

GPT-6 Astra spurs research interest in looped transformers and hidden reasoning

A new analysis examines the ideas behind GPT-6 Astra, focusing on transformer architectures that reuse the same blocks across multiple passes rather than adding more layers. It reviews recent work on recurrent depth, where looping blocks can increase effective model depth and enable internal computation that is not exposed in the visible output. The piece also discusses hidden chains of thought and what this implies for interpreting model reasoning.

Hugging Face releases beginner tutorial on training a Decision Transformer

Hugging Face published a walkthrough aimed at newcomers that covers training a Decision Transformer, a model that treats reinforcement learning as a sequence-prediction problem. The guide walks through setup, training, and evaluation steps so readers can build and test their own version. It is part of the platform's educational content for practitioners getting started with decision-making models.

WHY IT MATTERS ↘Lowering the entry barrier to Decision Transformers gives more teams a practical route into sequence-modeled reinforcement learning without building infrastructure from scratch, which could accelerate experimentation and shift competitive pressure toward data quality and evaluation rather than model access. It also broadens the base of practitioners able to audit and deploy decision-making models, an area where governance concerns remain largely unresolved.