A new arXiv paper presents a transformer-based framework that analyzes EEG signals converted into spectrogram form to identify schizophrenia. The approach aims to support diagnosis, which currently relies mainly on clinical evaluation, by using a non-invasive brain-activity measurement. The work is a preprint and has not yet been validated in clinical settings.
A new arXiv paper proposes a transformer-based meta-controller for adaptive control of robotic manipulators that must cope with friction memory they cannot directly observe. Existing attention-based controllers fix the number of attention heads before training and rely on expensive offline tuning; this work instead adjusts capacity incrementally at runtime. The authors position the method as a reinforcement-learning approach to adaptive control in a cross-listed machine learning submission.
A new arXiv paper proposes Bypass Observation, an architecture that adds read-only observation heads to selected Transformer layers so internal hidden states can be inspected without altering the model's behavior. The approach aims to bridge the gap between the high-dimensional reasoning space of large language models and the text-only outputs users normally see. It is presented as a conceptual design for non-intrusive semantic extraction.
Researchers introduce CONVTRAP-TN, a transformer-based architecture designed to determine whether two speakers share a first-order family relationship using only audio. The work also releases a new uncontrolled Mandarin kinship speech dataset and reports an ablation study on the model's components. Kinship verification from voice is a relatively underexplored area, and the Mandarin dataset addresses a gap in multilingual resources.
A new arXiv paper introduces methods to measure how large language models store and encode contextual information. The authors report that tokens usually treated as trivial, such as determiners and punctuation marks, hold unexpectedly high contextual weight, and that removing them measurably affects model behavior. The work offers a way to probe internal context retention in transformers.
A new arXiv paper proposes an "attention bridge" technique for converting pretrained Transformer models into Mamba-style state-space models. The approach aims to make the distillation process more data efficient, addressing the high compute cost of training competitive SSMs from scratch. The work targets the gap between the mature Transformer ecosystem and the less developed tooling around state-space architectures.
A new arXiv preprint introduces FlowTSFM, an approach for encoder-based time series foundation models that assigns a predictive role to intermediate Transformer layers instead of supervising only the final forecast. The method recasts encoder depth as a form of quantile transport, according to the abstract. The announcement provides only the opening portion of the paper's abstract, so full details of the architecture and evaluation are not yet available in this report.
A new arXiv preprint examines Delayed Tensor Parallelism (DTP), a scheme that avoids the blocking all-reduce step in tensor-parallel Transformer inference by letting each device immediately add its partial output to the residual stream and broadcast it. The authors introduce affinity-aware sharding as a way to organize this communication and computation across devices. The work targets more efficient distributed inference for large Transformer models.
A new arXiv paper examines how different tokenization choices affect ECG transformer models, since the tokenizer decides both the physiological signal content the model sees and the sequence length attention operates over. The authors benchmark eight tokenization strategies across four architectures — Transformer, Informer, Reformer, and FEDformer — on the nine-label CPSC ECG dataset. The work is cross-listed in cs.AI and cs.LG.
A new arXiv preprint examines how recurrent transformers that reuse a single set of weights differ from conventional stacked transformers in representation and dynamics. Using 12-lead electrocardiogram data as the test domain, the authors analyze the mechanistic differences between the two designs. The work aims to clarify the trade-offs behind the parameter efficiency that has made looped architectures popular.
Researchers propose BiHDTrans, a transformer variant that uses binary hyperdimensional computing to classify multivariate time series from IoT sensors. The design targets resource-constrained edge devices, where large data volumes and limited compute make standard models impractical. It is presented as an arXiv preprint focused on balancing efficiency with classification accuracy.
A new arXiv paper introduces contrastive projection, a method for inspecting transformer hidden states that works by differencing logit lens outputs. The authors argue that a standard logit lens reading of a single intermediate-layer state is dominated by generic tokens a model would predict for nearly any input, making such readings unreliable. Subtracting these baseline predictions isolates the input-specific component of the internal state, offering a more trustworthy window into model internals.
A new arXiv paper proposes a technique for measuring how logically consistent transformer language models are during multi-step reasoning, using analysis of query-key alignment in attention mechanisms. The authors position this as a way to assess reasoning reliability beyond improvements from prompting methods such as Chain-of-Thought. The work appears on arXiv under cs.AI (2502.17017).
A new arXiv paper introduces Elastoformer, a method that reshapes transformer models at runtime using elastic transformations so they can adjust to shifting operating conditions. The work targets computer vision workloads on edge devices, where latency, power, and compute budgets fluctuate. By adapting model capacity on the fly, the approach aims to let on-device systems balance accuracy against resource constraints in real time.
A new arXiv paper proposes EFQ-Softmax, a technique for quantizing the softmax step in Transformer attention without computing exponentials. Low-bit attention already speeds up inference by shifting matrix multiplications to FP8 or FP4 engines, but the softmax path often still needs higher precision for shifted-score exponentials and temporary probability storage. The work targets that remaining bottleneck in the attention pipeline.