A revised arXiv preprint introduces "coupled scaling," a framework that explains when two learning systems trained on the same task under identical resource constraints should exhibit matching scaling rates and when they should diverge. The approach ties scaling behavior to representational accessibility, describing how easily a model can reach task-relevant features.
A new arXiv preprint examines how learning rate schedules can be optimized within the functional scaling law framework, which separates training dynamics into signal learning and noise forgetting. The authors analyze power-law kernel regression to characterize these two components, comparing schedules such as power decay and warmup-stable-decay. The work offers theoretical guidance on choosing learning rate schedules for model training.
A new arXiv preprint examines grokking, the phenomenon where neural networks keep memorizing training data before abruptly improving on held-out data. While prior work has focused on why this delay happens, the paper targets its quantitative structure, describing scaling laws and a phase structure that predict when the shift occurs. The authors present an arXiv preprint; the abstract excerpt provided does not detail the full experimental setup or results.