Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization (2024.findings-emnlp)
Copied to clipboard
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, Min Zhang
| Challenge: | Large language models require a balance between efficiency and performance. |
| Approach: | They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici. |
| Outcome: | The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio. |
Similar Papers
1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank approximation compresses the model by retaining its essential structure with minimal information loss. |
| Approach: | They propose a method that leverages the strengths of pruning and low-rank approximation for LLMs. |
| Outcome: | The proposed methods surpass the existing methods on LLaMA and Qwen2.5 models. |
Structured Pruning of Large Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to remarkable improvements on a variety of tasks. |
| Approach: | They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training. |
| Outcome: | The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference. |
FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression (2026.findings-eacl)
Copied to clipboard
| Challenge: | Low-rank decomposition methods suffer from accuracy degradation and expensive calibration procedures. |
| Approach: | They propose a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. |
| Outcome: | The proposed method outperforms pruning baselines in generalization and downstream performance while delivering inference speedups. |
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank compression can reduce memory usage and computational demand, but results are poor during decoding. |
| Approach: | They propose a fine-grained low-rank compression algorithm that determines optimal rank allocation for each layer and incorporates progressive low-ranked decoding to maintain text generation quality. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on summarization tasks and on understanding tasks. |
Surgical Feature-Space Decomposition of LLMs: Why, When and How? (2024.acl-long)
Copied to clipboard
| Challenge: | Low-rank approximations of the weight and feature space can enhance the performance of large language models. |
| Approach: | They propose to use weight and feature space decomposition to improve LLM performance . they also extend their investigation to the implications of low-rank approximations on model bias . |
| Outcome: | The proposed low-rank approximations can improve performance of large language models . the authors show that the approximate can improve generalization and inference performance . |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs. |
| Approach: | They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. |
| Outcome: | The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics. |
RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model (2025.findings-naacl)
Copied to clipboard
| Challenge: | Current compression techniques entail structural pruning and a recovery phase that leverages the Low-Rank Adaptation algorithm. |
| Approach: | They propose a hierarchical rank allocation method that enables efficient fine-tuning of pruned LLMs according to layerwise specific recovery requirements. |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods across pruning settings and LLM architectures with improvements ranging from 0.7% to 5.5%. |
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. |
| Approach: | They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency. |
| Outcome: | The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B. |