| Challenge: | Existing attempts to integrate sparsification with Mamba fail to leverage Mamba's internal structure for fine-grained sparsifying. |
| Approach: | They propose to use Mamba to integrate sparsification into Mamba and propose a flexible and effective mechanism for parameter scalability. |
| Outcome: | The proposed framework can independently achieve parameter scalability and has stronger performance. |
Similar Papers
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained models have achieved outstanding results in sequence modeling . alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address these inefficiencies. |
| Approach: | They propose to reduce the size and computational overhead of large pre-trained models by removing selected components at different granularities. |
| Outcome: | The proposed models achieve a speedup of up to 1.4x during inference while maintaining accuracy. |
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-space models struggle with quadratic computational complexity, limiting their use in long-context tasks and resource-constrained input data. |
| Approach: | They propose a pruning framework specifically tailored for Mamba that reduces parameter counts by 70% with only a 3–9% drop in performance. |
| Outcome: | The proposed pruning framework achieves up to 70% parameter reduction with only a 3–9% drop in performance. |
Q-Mamba: Towards more efficient Mamba models via post-training quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that Mamba architectures have room for further optimization in linear projections and state caches. |
| Approach: | They propose a decoupled scale quantization scheme to mitigate outliers in states and channels by applying separate quantization scales. |
| Outcome: | The proposed method reduces memory consumption by 50% across various quantization settings, model sizes, and generation and zero-shot tasks. |
ReMamba: Equip Mamba with Effective Long-Sequence Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mamba models demonstrate superior inference efficiency and competitive performance on short-context tasks, but their capacity to comprehend long contexts is limited compared to transformer-based models. |
| Approach: | They propose a model which incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. |
| Outcome: | The proposed model improves on the LongBench and L-Eval benchmarks by 3.2 and 1.6 points and attains performance almost on par with same-size transformer models. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models (2025.acl-long)
Copied to clipboard
| Challenge: | State space models lack interpretability tools for long-context sequence modeling. |
| Approach: | They propose a token-level decomposition method for Mamba-1 and Mamba-2 that enables fine-grained interpretability. |
| Outcome: | The proposed method is able to reveal Mamba’s token-to-token interaction patterns across multiple tasks including translation, copying, and retrieval-based generation. |
Mamba Knockout for Unraveling Factual Information Flow (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has introduced Mamba-based SSM architectures that rival Transformer performance in various settings. |
| Approach: | They propose to use attentional interpretability techniques originally developed for Transformers to trace how information is transmitted and localized across tokens and layers. |
| Outcome: | The proposed model disentangles how distinct features enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations. |
Mamba Drafters for Speculative Decoding (2025.findings-emnlp)
Copied to clipboard
Daewon Choi, Seunghyuk Oh, Saket Dingliwal, Jihoon Tack, Kyuyoung Kim, Woomin Song, Seojin Kim, Insu Han, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, Sravan Babu Bodapati
| Challenge: | Existing drafters that use external drafters suffer from slower drafting while self-speculation methods use drafters tailored to the target model but require re-training. |
| Approach: | They propose a drafter based on a state space model, Mamba, as a solution that combines the best aspects of both approaches. |
| Outcome: | The proposed drafters outperform existing drafters while using less memory and maintaining their cross-model adaptability. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
Exploring the Limitations of Mamba in COPY and CoT Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Inference overhead of Transformers increases linearly with the sequence length, posing challenges for modeling long sequences. |
| Approach: | They analyze Mamba's expressive ability to perform COPY operations and Chain of Thought reasoning tasks using a defined sequence length. |
| Outcome: | The proposed model can perform COPY operations and Chain of Thought reasoning tasks with a constant size while reducing computational costs. |