As easy as PIE: understanding when pruning causes language models to disagree (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language Model pruning reduces the model's efficiency by removing weights, nodes, or other parts of its architecture. |
| Approach: | They propose to prune Language Models (LMs) to produce smaller, hence more efficient models with small loss to their effectiveness. |
| Outcome: | The proposed pruning method hurts data points that matter the most when pruning . the proposed pruning technique is based on a new study of NLP datasets . |
Similar Papers
On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language model pruning have shown high predictive performance in post-training settings. |
| Approach: | They conduct an empirical study on the performance and internal representation changes associated with pruning multilingual models for monolingual applications. |
| Outcome: | The proposed pruning methods retain perplexity and yield high signal-to-noise ratios, but not consistently improve downstream tasks. |
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization (2024.emnlp-main)
Copied to clipboard
| Challenge: | minimizing reconstruction error is not always ideal and can overfit calibration data. |
| Approach: | They propose a method to prune large language models by divide and conquer . they propose minimizing reconstruction error by more than 90% by using calibration data . |
| Outcome: | The proposed pruning approach generates high reconstruction errors . the proposed technique reduces reconstruction error by more than 90% . |
On the Impact of Calibration Data in Post-training Quantization and Pruning (2024.acl-long)
Copied to clipboard
| Challenge: | Quantization and pruning are the foundations of compression for large language models . however, no prior work has investigated how calibration data impacts performance of compression methods. |
| Approach: | They propose an empirical study on the effect of calibration data on LLM performance. |
| Outcome: | The proposed methods improve performance in a post-training setting. |
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pruning results on benchmark transformers, such as BERT, are not as remarkable as those of convolutional neural networks. |
| Approach: | They propose to apply a knowledge-aware pruning process to transformer-based pre-trained language models to reduce model size and model weight. |
| Outcome: | The proposed pruning method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions (2025.emnlp-main)
Copied to clipboard
| Challenge: | reducing the size of LLMs through post-training pruning has been studied, but its impact on model fairness remains unexplored. |
| Approach: | They propose a pruning method that removes parameters that are redundant for input processing but influential in output generation. |
| Outcome: | The proposed pruning method can maintain or improve fairness across models and tasks where existing methods have limitations. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
Structured Pruning of Large Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to remarkable improvements on a variety of tasks. |
| Approach: | They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training. |
| Outcome: | The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference. |
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes. |
| Approach: | They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques. |
| Outcome: | The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks. |