A new arXiv paper analyzes how representations inside transformer models change across layers, treating each learned update as a combination of a component that keeps the existing direction and one that shifts it elsewhere. The authors frame this as a functional geometry, aiming to explain what the model preserves versus reorients as information flows through the network.
An arXiv research paper questions why dense Transformers almost universally place most of their non-embedding parameters in narrow-wide-narrow feed-forward networks. Drawing on theoretical and empirical evidence, the authors explore an alternative wide-narrow-wide (hourglass) residual design for these blocks. The work is cross-listed across the cs.AI, cs.CL, and cs.LG categories on arXiv.
A cs.CL preprint argues that rotary position encoding gives Transformer attention a phase structure that typical vector-geometry analyses overlook. The authors develop a bounded spectral framework to study how queries and keys interact under this phase-based view. They apply it to semantic continuity and to governing where model execution begins and ends.