Weight-Aware Activation Sparsity with Constrained Bayesian Optimization Scheduling for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing activation sparsification methods rely on activation magnitude and weights for sparsity . authors propose a weight-aware activation-a-ware framework for large language models . |
| Approach: | They propose a weight-aware activation sparsity framework that uses weight-based scoring to measure activation importance in sparsification and a custom GPU sparse kernel to support it. |
| Outcome: | The proposed framework outperforms existing methods at 60% model-level sparsity and significantly outperfies them at higher sparsities. |
Similar Papers
Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression (2026.acl-long)
Copied to clipboard
Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang, Jun Wang
| Challenge: | Large language models (LLMs) excel at natural language tasks but face deployment bottlenecks due to computational demands. |
| Approach: | They propose a training-free framework that exploits activation and weight sparsity . they use a three-tier routing strategy that uses magnitude-based pruning . |
| Outcome: | Experiments on Llama and Mistral models show that DAWS outperforms activation-weight sparsity pruning methods. |
Motivating Next-Gen Accelerators with Flexible N:M Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches (2026.acl-industry)
Copied to clipboard
Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina, Vladimir Smirnov, Redko Dmitry, Alexey Dontsov, Maxim Zhelnin, Evgeny Burnaev, Egor Shvetsov
| Challenge: | Recent studies show that sparsification is not supported in large language models. |
| Approach: | They propose to use activation pruning to accelerate large language models with sparsification . they compare activation pruners with weight pruner and activater pruning with activation . |
| Outcome: | The proposed approach outperforms weight pruning at matched sparsity levels. |
CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for activation sparsification do not capture the relationship between activation and model performance. |
| Approach: | They propose a general activation sparsification approach using channel-wise thresholding and selective sparsifying to capture the relationship between activation and model performance. |
| Outcome: | The proposed approach reduces the number of activated neurons during inference by 1.27x over eight downstream tasks while activating fewer parameters than existing methods. |
Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing sparsity methods lack adaptivity to contextual or model structural demands or incur prohibitive computational overhead. |
| Approach: | They propose a Cognitive-Load-Aware Dynamic Activation framework that synergizes statistical sparsity with semantic adaptability. |
| Outcome: | The proposed framework achieves 20% average speedup with less than 2% accuracy degradation outperforming Griffin and TT. |
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models (2025.coling-main)
Copied to clipboard
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, Maosong Sun
| Challenge: | Activation sparsity is a promising paradigm for accelerating model inference . few large language models achieve high activation spar and comparable performance . |
| Approach: | They propose a method to achieve activation sparsity and acceleration in large language models . they introduce ReLU activation and adopt progressive sparse regularization . |
| Outcome: | The proposed method achieves high activation sparsity and comparable model performance. |
COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have created significant computational inefficiencies due to their size and complexity. |
| Approach: | They propose to use a linear combination to deactivate non-essential parameters during inference to reduce computational costs. |
| Outcome: | The proposed methods can omit 90% of computations with performance loss as low as 5.5% ideally, while M-COUNTDOWN provides a predictor-free solution with up to 29.4% better performance preservation compared to existing methods. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
IMPACT: Importance-Aware Activation Space Reconstruction (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) achieve strong performance across domains but remain difficult to deploy in resource-constrained environments due to their massive size. |
| Approach: | They propose an importance-aware activation reconstruction framework that links compression to its effect on model performance. |
| Outcome: | Experiments show that IMPACT reduces model size by 55.4% while maintaining accuracy comparable to or better than state-of-the-art models. |
ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yash Akhauri, Ahmed AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander Rush, Safeen Huda, Mohamed Abdelfattah
| Challenge: | Prior work has focused on contextual sparsity, but it has not been successful. |
| Approach: | They propose a novel pruning predictor that can shadow the LLM behavior and enforce better sparsity patterns. |
| Outcome: | The proposed model can shadow the LLM behavior and enforce better sparsity patterns, resulting in 15% improvement in end-to-end accuracy compared to prior methods. |