| Challenge: | In spite of impressive results of neural networks, the huge model size has hindered their applications in cases where computation and memory resources are limited. |
| Approach: | They propose a method for layer-wise pruning using mutual information based feature selection in SVMs and logistic regression. |
| Outcome: | The proposed pruning strategy offers greater speedup and higher performance than weight-based pruning methods. |
Similar Papers
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |
Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in processing long contexts. |
| Approach: | They propose a training-free method that adaptively chooses the selection layer for KV cache reduction . they exploit the variance of token ranks ordered by attention score to optimize decoding . |
| Outcome: | The proposed method outperforms state-of-the-art token pruning methods on InfiniteBench, RULER, and NIAH benchmarks. |
Structured Optimal Brain Pruning for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods for Large Language Models rely on unstructured pruning or require special hardware to accelerate computation. |
| Approach: | They propose a retraining-free structured pruning method called SoBP . they evaluate the effectiveness of SoBP across 14 models from 3 LLM families . |
| Outcome: | The proposed method outperforms current state-of-the-art pruning methods on 8 datasets. |
BlockPruner: Fine-grained Pruning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant computational and memory costs associated with training and inference. |
| Approach: | They propose a training-free structured pruning approach that targets redundancies in MHA and MLP blocks. |
| Outcome: | The proposed pruning approach achieves more granular and effective pruning compared to state-of-the-art pruning methods. |
GAP: a Global Adaptive Pruning Method for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing structured pruning methods employ uniform compression rates across network layers, neglecting the varying importance of different network depths. |
| Approach: | They propose a pruning framework that minimizes global capability loss by layer-adaptive pruning rates. |
| Outcome: | The proposed approach achieves comparable performance with state-of-the-art methods at high pruning rates and shows significant advantages at low pruning rates. |
Data Pruning for Efficient Model Pruning in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have demonstrated encouraging performance in various NLP tasks at the cost of over-parametrized networks and high memory requirements. |
| Approach: | They combine data pruning with movement pruning for Neural Machine Translation to enable efficient fine-pruning by leveraging cross-entropy scores of individual training instances. |
| Outcome: | The proposed pruning strategy outperforms other pruning methods on a translation task and shows that training cross-entropy scores can reduce the steps required for convergence and training time. |
Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Neural network pruning disrupts LLMs’ internal activation features crucial for lie detection . layer-wise pruning sparsity inadvertently removes crucial weights, failing to improve lie detection performance despite its reliance on the most crucial LLM layer. |
| Approach: | They propose a pruning approach that places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously. |
| Outcome: | The proposed approach improves the hallucination detection for pruned LLMs (achieving 88% accuracy at 50% sparsity) and enhances their performance on TruthfulQA. |
Aligned Weight Regularizers for Pruning Pretrained Neural Networks (2022.findings-acl)
Copied to clipboard
| Challenge: | Pruning aims to reduce the number of parameters while maintaining performance close to the original network. |
| Approach: | They propose a self-distilled pruning strategy that maximizes representational similarity between pruned and unpruned networks. |
| Outcome: | The proposed pruning strategy outperforms smaller models and outperformed smaller ones with an equal number of parameters and is competitive against (6 times) larger distilled networks. |
Structured Pruning for Diverse Best-of-N Reasoning Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on the MATH dataset demonstrate that our method significantly outperforms traditional best-of-N and random head selection strategies. |
| Approach: | They propose a contrastive learning framework that dynamically selects the optimal head and layer to prune during inference by aligning question embeddings with head embedds. |
| Outcome: | The proposed approach outperforms best-of-N and random head selection strategies on the MATH500 and GSM8K datasets. |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |