| Challenge: | Inference overhead of Transformers increases linearly with the sequence length, posing challenges for modeling long sequences. |
| Approach: | They analyze Mamba's expressive ability to perform COPY operations and Chain of Thought reasoning tasks using a defined sequence length. |
| Outcome: | The proposed model can perform COPY operations and Chain of Thought reasoning tasks with a constant size while reducing computational costs. |
Similar Papers
ReMamba: Equip Mamba with Effective Long-Sequence Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mamba models demonstrate superior inference efficiency and competitive performance on short-context tasks, but their capacity to comprehend long contexts is limited compared to transformer-based models. |
| Approach: | They propose a model which incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. |
| Outcome: | The proposed model improves on the LongBench and L-Eval benchmarks by 3.2 and 1.6 points and attains performance almost on par with same-size transformer models. |
Recall with Reasoning: Chain-of-Thought Distillation for Mamba’s Long-Context Memory and Extrapolation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context memory methods such as Mamba struggle with long-constituency when the length of the processed text exceeds the model's training length. |
| Approach: | They propose a method that uses chain-of-thought summarization to teach Mamba to actively recall and reason over long contexts. |
| Outcome: | Experiments on LONGMEMEVAL and HELMET show that RwR outperforms existing long-term memory methods while preserving short-context capabilities. |
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that Chain-of-thought (CoT) can enhance the performance of large language models (LLMs) however, there is limited understanding of the algorithms that Transformer+CoT can learn. |
| Approach: | They propose two metrics to evaluate Transformer+CoT's state tracking capabilities and identify the circuit responsible for tracking the world state. |
| Outcome: | The proposed model achieves 100% accuracy for each state, highlighting an implicit finite state automaton (FSA) embedded within the model. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained models have achieved outstanding results in sequence modeling . alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address these inefficiencies. |
| Approach: | They propose to reduce the size and computational overhead of large pre-trained models by removing selected components at different granularities. |
| Outcome: | The proposed models achieve a speedup of up to 1.4x during inference while maintaining accuracy. |
Revealing and Mitigating the Local Pattern Shortcuts of Mamba (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Mamba excels in tasks that involve localized key information but faces challenges with tasks that require handling distributed key information. |
| Approach: | They propose to introduce a global gate module into Mamba to address this problem by adding 4M extra parameters to the model. |
| Outcome: | The proposed model outperforms attention-based models on synthetic and synthetic tasks with only 4M extra parameters. |
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing theoretical treatments have shed light on some aspects of CoT, but we are still far from a concrete understanding of the concept. |
| Approach: | They propose to formalize CoT reasoning in a probabilistic setting to bridge this gap . they show that LMs can represent the same family of distributions over strings as probabilistic Turing machines. |
| Outcome: | The proposed model can represent the same family of distributions over strings as Turing machines. |
The Hidden Attention of Mamba Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that Mamba models can be used for multiple domains, including NLP, long-range sequence processing, and computer vision. |
| Approach: | They add a third view and show that Mamba models can be viewed as attention-driven models. |
| Outcome: | The proposed model can be viewed as attention-driven and empirically compare it to the attention-based models of transformers. |
An Exploration of Mamba for Speech Self-Supervised Models (2026.acl-long)
Copied to clipboard
Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee
| Challenge: | Mamba-based SSL models are promising for long-sequence modeling, speech unit extraction, and speech self-supervised learning. |
| Approach: | They propose to use Mamba-based HuBERT models as an alternative to Transformer-based SSL architectures. |
| Outcome: | The proposed models outperform Transformer-based models in language modeling tasks while showing superior performance on streaming ASR. |
Mamba Knockout for Unraveling Factual Information Flow (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has introduced Mamba-based SSM architectures that rival Transformer performance in various settings. |
| Approach: | They propose to use attentional interpretability techniques originally developed for Transformers to trace how information is transmitted and localized across tokens and layers. |
| Outcome: | The proposed model disentangles how distinct features enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations. |