NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurelie Lozano, Payel Das, Georgios Kollias
| Challenge: | Transformer-based Language Models have become ubiquitous in natural language processing due to impressive performance on various tasks. |
| Approach: | They explore how sparsity affects network topology by exploiting mechanisms seen in biological networks . they show that model-agnostic sparsities are performant across diverse NLP tasks . |
| Outcome: | The proposed model-agnostic sparsity approaches are performant and efficient across NLP tasks. |
Similar Papers
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yash Akhauri, Ahmed AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander Rush, Safeen Huda, Mohamed Abdelfattah
| Challenge: | Prior work has focused on contextual sparsity, but it has not been successful. |
| Approach: | They propose a novel pruning predictor that can shadow the LLM behavior and enforce better sparsity patterns. |
| Outcome: | The proposed model can shadow the LLM behavior and enforce better sparsity patterns, resulting in 15% improvement in end-to-end accuracy compared to prior methods. |
Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing sparsity methods lack adaptivity to contextual or model structural demands or incur prohibitive computational overhead. |
| Approach: | They propose a Cognitive-Load-Aware Dynamic Activation framework that synergizes statistical sparsity with semantic adaptability. |
| Outcome: | The proposed framework achieves 20% average speedup with less than 2% accuracy degradation outperforming Griffin and TT. |
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models (2022.acl-demo)
Copied to clipboard
| Challenge: | Large pre-trained language models have been used for many NLP tasks but computational resources are limited. |
| Approach: | They propose an open-source model pruning toolkit for pre-trained language models . they propose a self-supervised pruning method that can be applied without labeled data. |
| Outcome: | The proposed pruning method reduces model size without retraining the model and speeds up inference speed on the common CPU and GPU devices. |
Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression (2026.acl-long)
Copied to clipboard
Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang, Jun Wang
| Challenge: | Large language models (LLMs) excel at natural language tasks but face deployment bottlenecks due to computational demands. |
| Approach: | They propose a training-free framework that exploits activation and weight sparsity . they use a three-tier routing strategy that uses magnitude-based pruning . |
| Outcome: | Experiments on Llama and Mistral models show that DAWS outperforms activation-weight sparsity pruning methods. |
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pruning results on benchmark transformers, such as BERT, are not as remarkable as those of convolutional neural networks. |
| Approach: | They propose to apply a knowledge-aware pruning process to transformer-based pre-trained language models to reduce model size and model weight. |
| Outcome: | The proposed pruning method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy. |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
BlockPruner: Fine-grained Pruning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant computational and memory costs associated with training and inference. |
| Approach: | They propose a training-free structured pruning approach that targets redundancies in MHA and MLP blocks. |
| Outcome: | The proposed pruning approach achieves more granular and effective pruning compared to state-of-the-art pruning methods. |
Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing pruning algorithms suffer from limitations such as architecture specificity and reliance on demanding calculations. |
| Approach: | They propose a pruning algorithm based on Kernel Density Estimation . it preserves most significant parameters while restoring others to their pre-training state . |
| Outcome: | The proposed pruning algorithm achieves better performance than the original unpruned version. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |