A new arXiv paper argues that truncating each weight matrix independently with SVD, while optimal per matrix, lets compression errors accumulate across a transformer block. The authors propose a three-level optimization scheme that accounts for how these errors compound through nonlinear layers. The work targets better accuracy retention in low-rank LLM compression.
A new arXiv paper introduces UniRank, a method for deciding how much rank to assign to each weight matrix when compressing large language models via low-rank decomposition. The authors argue that uniform or hand-tuned allocation schemes overlook differences in module importance. The work is released as a revised submission on arXiv.
A new preprint presents an optimization framework that pairs block tensor train decompositions with the Burer-Monteiro approach to make low-rank quantum state tomography more computationally tractable. Reconstructing quantum states from measurement data is essential for evaluating quantum devices, but conventional estimators scale poorly. The work appeared as a cross-listed arXiv paper in the machine learning category.
A new arXiv paper examines how the way post-training updates are parameterized influences the length of text that large language models produce. The authors propose LOCUS, a task-aware low-rank adaptation method intended to curb the verbosity that standard preference alignment tends to introduce without sacrificing usefulness. The work targets serving costs, which grow with output length.