| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
Similar Papers
Token Pruning for Improving Graph-Generating State Space Model Performance (2026.eacl-srw)
Copied to clipboard
| Challenge: | State Space Models (SSMs) are efficient alternatives to Transformers for sequence modeling, but extending them to two-dimensional vision tasks remains challenging. |
| Approach: | They propose a leaf-guided pruning strategy that accelerates GG-SSM inference . they selectively scales or bypasses secondary refinement computations associated with leaf nodes . |
| Outcome: | The proposed pruning strategy accelerates GG-SSM inference without modifying graph topology . the proposed pruning method achieves throughput improvements with controlled accuracy degradation . |
State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models (2025.acl-short)
Copied to clipboard
| Challenge: | State Space Models (SSMs) have emerged as efficient alternatives to Transformers, but their application to SSMs remains unexplored. |
| Approach: | They propose a state-based PEFT method that adjusts state directly instead of using external prompts. |
| Outcome: | The proposed method is based on state-offset tuning, which directly affects state at every timestep. |
CodeSSM: Towards State Space Models for Code Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformers have limitations, such as quadratic complexity and high inference costs. |
| Approach: | They propose a state space model that is trained on code corpora to assess its effectiveness. |
| Outcome: | The proposed model reduces memory usage by up to 64% compared to transformers at a context length of 2048. |
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-space models struggle with quadratic computational complexity, limiting their use in long-context tasks and resource-constrained input data. |
| Approach: | They propose a pruning framework specifically tailored for Mamba that reduces parameter counts by 70% with only a 3–9% drop in performance. |
| Outcome: | The proposed pruning framework achieves up to 70% parameter reduction with only a 3–9% drop in performance. |
DenseSSM: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) face excessive computational and memory requirements due to the commonly used Transformer architecture. |
| Approach: | They propose a method to enhance the flow of hidden information between layers in large language models by selectively integrating shallow-layer hidden states into deeper layers. |
| Outcome: | The proposed method maintains parallelizability and inference efficiency of SSMs while significantly boosting performance on public benchmarks. |
Rethinking Token Reduction for State Space Models (2024.emnlp-main)
Copied to clipboard
Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, Yanzhi Wang
| Challenge: | Existing methods for token reduction for SSMs lead to performance drops . a recent study shows that Mamba-2 improves the accuracy of the model by 5.7% to 13.1% . |
| Approach: | They propose a token reduction method that integrates token importance and similarity into SSMs and takes advantage of pruning and merging. |
| Outcome: | The proposed method improves accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods while reducing computational demands and memory requirements. |
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization (2024.emnlp-main)
Copied to clipboard
| Challenge: | minimizing reconstruction error is not always ideal and can overfit calibration data. |
| Approach: | They propose a method to prune large language models by divide and conquer . they propose minimizing reconstruction error by more than 90% by using calibration data . |
| Outcome: | The proposed pruning approach generates high reconstruction errors . the proposed technique reduces reconstruction error by more than 90% . |
Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |
Two-Stage Regularization-Based Structured Pruning for LLMs (2026.acl-long)
Copied to clipboard
Mingkuan Feng, Jinyang Wu, Siyuan Liu, Shuai Zhang, Hongjian Fang, Ruihan Jin, Feihu Che, Pengpeng Shao, Zhengqi Wen, Jianhua Tao
| Challenge: | Structural pruning is a promising solution for large language models . prior structured pruning methods remove unimportant parameters based on certain metrics . |
| Approach: | They propose a structural pruning method that iteratively learns the weights of transformer layers by adding their l1-norm to the loss function. |
| Outcome: | The proposed pruning method outperforms strong layer-wise pruning methods without requiring retraining. |
BlockPruner: Fine-grained Pruning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant computational and memory costs associated with training and inference. |
| Approach: | They propose a training-free structured pruning approach that targets redundancies in MHA and MLP blocks. |
| Outcome: | The proposed pruning approach achieves more granular and effective pruning compared to state-of-the-art pruning methods. |