A new arXiv paper argues that multilingual LLMs suffer from using one shared vocabulary across all supported languages, which produces uneven compression rates between languages. The authors also note that large embedding and output matrices raise memory demands and slow processing. Their proposed modular tokenizer design assigns separate tokenization components per language to address both issues.
A new arXiv paper examines how different tokenization choices affect ECG transformer models, since the tokenizer decides both the physiological signal content the model sees and the sequence length attention operates over. The authors benchmark eight tokenization strategies across four architectures — Transformer, Informer, Reformer, and FEDformer — on the nine-label CPSC ECG dataset. The work is cross-listed in cs.AI and cs.LG.
A new arXiv paper introduces methods to measure how large language models store and encode contextual information. The authors report that tokens usually treated as trivial, such as determiners and punctuation marks, hold unexpectedly high contextual weight, and that removing them measurably affects model behavior. The work offers a way to probe internal context retention in transformers.
A new arXiv preprint argues that tokenization remains a fixed, hand-designed compression step in large language model pipelines even as other components become trainable end-to-end. The authors report that reinforcement learning can be used to learn tokenization jointly with the model, with earlier work showing promise at scale. The paper appears as a cross-listed replacement submission in cs.AI and cs.LG.
Researchers present DuoTok, a tokenization method that separates vocal and accompaniment streams while keeping source information intact for multi-track music generation. The approach aims to balance acoustic detail, sequence modelability, and cross-track structure that existing codecs trade off against each other. A revised version of the preprint is now available.
Researchers introduce StreamAlign, a speech tokenization method that maps audio into tokens aligned with large language model token spaces while operating in a streaming, low-latency manner. Unlike existing text-aligned tokenizers that depend on offline automatic speech recognition, the approach addresses the latency and alignment limitations that offline processing imposes, enabling more efficient use of pretrained LLMs for speech tasks.