Outline of the training process for learning SVD ranks (click to enlarge).

Compressing a language model with low-rank decomposition means choosing a rank for every layer — too aggressive and accuracy collapses, too conservative and you save nothing. Existing methods either search a small set of heuristic candidates, or learn ranks with gradient descent but need expensive fine-tuning afterwards to recover performance. We propose LLRC (Learning to Low-Rank Compress): the model is frozen, and only a lightweight differentiable mask over each layer’s singular values is trained, so every layer discovers its own compression rate directly — with no fine-tuning after compression.