A new arXiv paper examines whether drifting models, which learn a gradual transport process during training but generate samples in a single step, can converge quickly to a target distribution. The authors identify obstructions to fast convergence at fixed scale and propose a multihead approach that improves convergence rates. The work is theoretical, focused on the training dynamics rather than a deployed system.
Researchers asked whether retrieval circuits that small Transformers learn with the help of soft positional priors keep functioning once that prior is taken away. They tested this using a model whose prior-based attention biases can be gradually annealed out during training. The results indicate that the specific training trajectory, not just the architecture, decides whether a learned circuit can stand on its own after the prior is removed.
A new arXiv paper proposes analyzing how deep neural networks learn by studying them at the level of functions rather than individual parameters. The authors borrow tools from statistical mechanics, treating parameter configurations as microscopic states to explain why networks with billions of weights show consistent, predictable learning patterns. The approach aims to provide a theoretical framework for understanding training dynamics in very large models.