Looped GPT-BERT Shows Small Language Models Can Trade Parameters for Computation

Researchers examined looped variants of GPT-BERT for the BabyLM 2026 shared task, in which a compact set of layers is executed repeatedly rather than stacking many distinct ones. Their findings suggest that reapplying a small parameter budget can match the performance of larger models when training data is scarce. The work positions recurrence as a compute-for-parameters trade-off for building efficient low-resource language models.