1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank approximation compresses the model by retaining its essential structure with minimal information loss. |
| Approach: | They propose a method that leverages the strengths of pruning and low-rank approximation for LLMs. |
| Outcome: | The proposed methods surpass the existing methods on LLaMA and Qwen2.5 models. |
Similar Papers
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization (2024.findings-emnlp)
Copied to clipboard
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Kehai Chen, Min Zhang
| Challenge: | Large language models require a balance between efficiency and performance. |
| Approach: | They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici. |
| Outcome: | The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank compression can reduce memory usage and computational demand, but results are poor during decoding. |
| Approach: | They propose a fine-grained low-rank compression algorithm that determines optimal rank allocation for each layer and incorporates progressive low-ranked decoding to maintain text generation quality. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on summarization tasks and on understanding tasks. |
FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression (2026.findings-eacl)
Copied to clipboard
| Challenge: | Low-rank decomposition methods suffer from accuracy degradation and expensive calibration procedures. |
| Approach: | They propose a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. |
| Outcome: | The proposed method outperforms pruning baselines in generalization and downstream performance while delivering inference speedups. |
Break Through the Compression Bottleneck: From Theory to Practice (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression methods suffer from bottleneck issues when compression ratio is increased. |
| Approach: | They propose a novel approach to combine low-rank decomposition and quantization methods to reduce the compression bottleneck. |
| Outcome: | The proposed method reduces the computational and memory overhead of existing methods while maintaining model accuracy. |
Comparing Text Compression Capabilities of Large Language Models with Traditional Compression Algorithms (2026.eacl-srw)
Copied to clipboard
| Challenge: | Experimental results show that large language models outperform baselines on non-English datasets . traditional methods remained dataset-agnostic, and the results suggest that current methods are impractical for the compression task. |
| Approach: | They evaluate the non-English and unstructured text compression performance of Large Language Models . they compare them with traditional baselines on datasets from eight most widely spoken languages . |
| Outcome: | The evaluated LLM outperformed baselines on non-English datasets . the results show that the current methods are highly impractical for the compression task . |
Structured Pruning of Large Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to remarkable improvements on a variety of tasks. |
| Approach: | They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training. |
| Outcome: | The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. |
| Approach: | They propose a Data and Model Compression Framework that categorizes data filtering methodologies into three distinct paradigms: (1) distribution-aware methods, (2) quality-a aware methods, and (3) hybrid approaches considering both dimensions. |
| Outcome: | The proposed framework can select the optimal LLM while saving approximately 20-fold in training time. |