Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
Similar Papers
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
Structured Pruning of Large Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to remarkable improvements on a variety of tasks. |
| Approach: | They propose a generic, structured pruning approach by parameterizing each weight matrix and adaptively removing rank-1 components during training. |
| Outcome: | The proposed method outperforms unstructured pruning and block pruning on language modeling tasks while achieving speedups during training and inference. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing efforts to train pre-trained language models have brought significant improvements to various NLP applications. |
| Approach: | They propose to compress bulky LMs while preserving useful information for a specific task. |
| Outcome: | The proposed method can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. |
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language model pruning have shown high predictive performance in post-training settings. |
| Approach: | They conduct an empirical study on the performance and internal representation changes associated with pruning multilingual models for monolingual applications. |
| Outcome: | The proposed pruning methods retain perplexity and yield high signal-to-noise ratios, but not consistently improve downstream tasks. |
Pruning Pre-trained Language Models Without Fine-Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to prune Pre-trained Language Models (PLMs) are overparameterized and require fine-tuning. |
| Approach: | They propose a pruning method that uses first-order pruning to prune PLMs while fine-tuning the remaining weights. |
| Outcome: | The proposed method outperforms first-order pruning and zero-order methods at sparsity levels. |
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models (2022.acl-demo)
Copied to clipboard
| Challenge: | Large pre-trained language models have been used for many NLP tasks but computational resources are limited. |
| Approach: | They propose an open-source model pruning toolkit for pre-trained language models . they propose a self-supervised pruning method that can be applied without labeled data. |
| Outcome: | The proposed pruning method reduces model size without retraining the model and speeds up inference speed on the common CPU and GPU devices. |
Pruning Multilingual Large Language Models for Multilingual Inference (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual large language models (MLLMs) demonstrate better zeroshot learning performance in non-English languages compared to large language model trained on English-dominant data. |
| Approach: | They propose a pruning approach to prune large language models using bilingual sentence pairs from English and other languages to enhance their performance in non-English language. |
| Outcome: | The proposed pruning strategy enhances the MLLMs’ performance in non-English language. |