A new arXiv paper argues that multilingual LLMs suffer from using one shared vocabulary across all supported languages, which produces uneven compression rates between languages. The authors also note that large embedding and output matrices raise memory demands and slow processing. Their proposed modular tokenizer design assigns separate tokenization components per language to address both issues.
A new arXiv preprint proposes MoME, a technique that combines conditional memory — token-indexed embedding tables that give a model cheap parametric lookups — with sparse capacity ideas inspired by Mixture-of-Experts. The method aims to make those lookups context-aware by routing them selectively, as part of broader efforts to scale language models more efficiently.
A new arXiv paper examines how the way post-training updates are parameterized influences the length of text that large language models produce. The authors propose LOCUS, a task-aware low-rank adaptation method intended to curb the verbosity that standard preference alignment tends to introduce without sacrificing usefulness. The work targets serving costs, which grow with output length.