Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pruning methods focus on a single pruning criterion and lack variety. |
| Approach: | They propose a model pruning strategy that generates several pruning masks randomly and then chooses the optimal mask from the pool of mask candidates. |
| Outcome: | The proposed pruning strategy achieves state-of-the-art performance across eight datasets from GLUE, particularly excelling at high levels of sparsity. |
Similar Papers
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing language models from unwanted behaviors in natural language understanding datasets is a topic with increasing interest in the NLP community. |
| Approach: | They propose a method to debiase language models from unwanted behaviors in NLU tasks by identifying pruning masks that can be applied to a finetuned model. |
| Outcome: | The proposed method shows superior performance and performance over standard methods. |
IG-Pruning: Input-Guided Block Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for efficient inference rely on fixed block masks, which can lead to suboptimal performance. |
| Approach: | They propose an input-aware block-wise pruning method that dynamically selects layer masks at inference time. |
| Outcome: | The proposed method outperforms state-of-the-art static depth pruning methods . it is particularly suitable for resource-constrained deployment scenarios . |
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Attribution methods assess the contribution of inputs to the model prediction. |
| Approach: | They propose a method which removes subsets of inputs and a model which is based on hidden layers to make the decision to include or disregard an input token. |
| Outcome: | The proposed method is efficient because it predicts rather than searches the inputs. |
Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient (2025.acl-long)
Copied to clipboard
| Challenge: | Recent pruning methods rely on heuristically hand-crafted metrics, leading to suboptimal performance. |
| Approach: | They propose a method that optimizes pruning masks by minimizing back-propagation . they learn an underlying Bernoulli distribution to sample binary pruning mask samples . |
| Outcome: | The proposed method is able to support global and heterogeneous pruning without back-propagation. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |
BlockPruner: Fine-grained Pruning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant computational and memory costs associated with training and inference. |
| Approach: | They propose a training-free structured pruning approach that targets redundancies in MHA and MLP blocks. |
| Outcome: | The proposed pruning approach achieves more granular and effective pruning compared to state-of-the-art pruning methods. |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
GAP: a Global Adaptive Pruning Method for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing structured pruning methods employ uniform compression rates across network layers, neglecting the varying importance of different network depths. |
| Approach: | They propose a pruning framework that minimizes global capability loss by layer-adaptive pruning rates. |
| Outcome: | The proposed approach achieves comparable performance with state-of-the-art methods at high pruning rates and shows significant advantages at low pruning rates. |
Disentangling the Roles of Representation and Selection in Data Pruning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for data pruning involve many different design choices, which have not been systematically studied. |
| Approach: | They decompose data pruning into two key components: data representation and selection algorithm . theoretical and empirical results highlight crucial role of representations . |
| Outcome: | The proposed method can be used to train models with less data. |
Structured Pruning Learns Compact and Accurate Models (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have high costs in terms of storage, memory, and computation time. |
| Approach: | They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency. |
| Outcome: | The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data. |