Papers with Pruning
Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing jailbreaking methods create adversarial prompts to bypass LLM safeguards. |
| Approach: | They propose a framework for generating stealthy jailbreak prompts that enables knowledge sharing across attack paths. |
| Outcome: | The proposed framework outperforms state-of-the-art methods for attacking both open and closed LLMs with attack success rates of >96%. |
Aligned Weight Regularizers for Pruning Pretrained Neural Networks (2022.findings-acl)
Copied to clipboard
| Challenge: | Pruning aims to reduce the number of parameters while maintaining performance close to the original network. |
| Approach: | They propose a self-distilled pruning strategy that maximizes representational similarity between pruned and unpruned networks. |
| Outcome: | The proposed pruning strategy outperforms smaller models and outperformed smaller ones with an equal number of parameters and is competitive against (6 times) larger distilled networks. |
P2 Law: Scaling Law for Post-Training After Model Pruning (2025.acl-long)
Copied to clipboard
| Challenge: | Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). |
| Approach: | They propose to use model pruning techniques to maintain high performance while reducing hardware requirements for large language models (LLMs). |
| Outcome: | The proposed model pruning law can be generalized to larger dataset sizes, larger model sizes, and higher pruning rates, offering valuable insights for resource allocation in pruned LLMs. |
FedSpaLLM: Federated Pruning of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing pruning methods assume public access to calibration data, which is impractical for privacy-sensitive applications. |
| Approach: | They propose a federated learning framework for pruning LLMs that prunes models locally based on private data while accounting for system heterogeneity and communication efficiency. |
| Outcome: | The proposed framework reduces communication overhead and personalizes pruning process based on client resources in federated settings. |
EBERT: Efficient BERT Inference with Dynamic Structured Pruning (2021.findings-acl)
Copied to clipboard
| Challenge: | Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. |
| Approach: | They propose a dynamic structured pruning algorithm that prunes model weights at run-time . they propose to prune the unimportant heads in multi-head self-attention layers . |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods on different tasks. |
CROP: Contextual Region-Oriented Visual Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing VLMs process entire images, leading to excessive visual tokens . redundant image information also introduces a large number of visual token, requiring much higher memory and computation in VLM. |
| Approach: | They propose a framework to prune visual tokens using localization and pruning . they propose CROP to locate local image regions relevant to the query . |
| Outcome: | The proposed framework outperforms existing visual token pruning methods on a wide range of tasks. |
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs . however, it is hindered by two critical challenges: substantial GPU memory consumption and low activated parameters. |
| Approach: | They propose an Expert-Selection Aware Compressor for Mixture-of-Experts (MoE) that aligns with the characteristics of MoE from the perspectives of quantization and pruning. |
| Outcome: | The proposed approach significantly reduces memory usage and improves inference speed with minimal performance degradation. |
Lightweight Haar Wavelet Subband Pruning for LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive performance but require computational and memory resources. |
| Approach: | They propose a post-training framework that uses a Haar wavelet transform to prune weights. |
| Outcome: | The proposed pruning framework reduces pruning time and computational costs by removing less important weights while preserving model architecture. |
Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Neural network pruning disrupts LLMs’ internal activation features crucial for lie detection . layer-wise pruning sparsity inadvertently removes crucial weights, failing to improve lie detection performance despite its reliance on the most crucial LLM layer. |
| Approach: | They propose a pruning approach that places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously. |
| Outcome: | The proposed approach improves the hallucination detection for pruned LLMs (achieving 88% accuracy at 50% sparsity) and enhances their performance on TruthfulQA. |
LaCo: Layer-wise Compensation for Pruned Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for predicting performance degradations of Large Language Models (LLMs) neglect the structural distortions caused by sparsity. |
| Approach: | They propose a framework that reorients the recovery paradigm from global adaptation to hierarchical representation alignment by sequentially optimizing each layer to reconstruct the model's hidden states. |
| Outcome: | The proposed framework surpasses parameter-efficient baselines in perplexity reduction and zero-shot reasoning. |
FISTAPruner: Layer-wise Post-training Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods require inefficient retraining for billion-scale LLMs or rely on heuristicically designed metrics to determine pruning masks, leading to performance degradation. |
| Approach: | They propose a convex optimization model that induces sparsity in large language models by leveraging FISTA. |
| Outcome: | The proposed method can remove 50% of model parameters while retaining 98.6% and 95.6% of the zero-shot performance. |