Papers with SSM
Modeling Emotion Dynamics in Song Lyrics with State Space Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies in music emotion recognition assume a single label for the whole song, but annotated data is scarce and difficult to obtain. |
| Approach: | They propose a method to predict emotion dynamics in song lyrics without annotation . they frame each song as a time series and use a State Space Model to generate the full emotion dynamics. |
| Outcome: | The proposed method improves performance of sentence-level baselines without annotating songs, making it ideal for limited training scenarios. |
Incorporating Exponential Smoothing into MLP: a Simple but Effective Sequence Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Structured State Space models (SSMs) have been used for long-range sequence learning but are limited in their complexity and computational and memory requirements. |
| Approach: | They propose to incorporate a simple SSM into an element-wise MLP to reduce inductive bias. |
| Outcome: | The proposed model achieves comparable results to existing models on the LRA benchmark. |
The Hidden Attention of Mamba Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that Mamba models can be used for multiple domains, including NLP, long-range sequence processing, and computer vision. |
| Approach: | They add a third view and show that Mamba models can be viewed as attention-driven models. |
| Outcome: | The proposed model can be viewed as attention-driven and empirically compare it to the attention-based models of transformers. |
FlowHN: Adaptive Token Routing for Efficient Parallel Hybrid Networks (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing hybrids lack performance, latency, and cost-efficient scaling for production LLMs. |
| Approach: | They propose a deployment-oriented parallel hybrid architecture that enables deterministic conditional computation via FLOP-aware token circulation across attention and SSM branches. |
| Outcome: | FlowHN achieves 4 higher throughput and 15% higher MFU than current models while maintaining competitive accuracy on reasoning, coding, and long-context tasks. |
Rethinking Token Reduction for State Space Models (2024.emnlp-main)
Copied to clipboard
Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, Yanzhi Wang
| Challenge: | Existing methods for token reduction for SSMs lead to performance drops . a recent study shows that Mamba-2 improves the accuracy of the model by 5.7% to 13.1% . |
| Approach: | They propose a token reduction method that integrates token importance and similarity into SSMs and takes advantage of pruning and merging. |
| Outcome: | The proposed method improves accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods while reducing computational demands and memory requirements. |
GL-GAN: Perceiving and Integrating Global and Local Styles for Handwritten Text Generation with Mamba (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models lack the ability to perceive and integrate handwriting styles, which affects the realism of the synthesized samples. |
| Approach: | They propose a Hybrid Style Encoder that captures global and local styles and integrates them into a Dynamic Feature Enhancement Module (DFEM). |
| Outcome: | The proposed model outperforms state-of-the-art models on two widely used handwriting datasets and can provide training data for handwritten text recognition and signature verification. |
Salient Span Masking for Temporal Understanding (2023.eacl-main)
Copied to clipboard
| Challenge: | Salient Span Masking (SSM) has shown to be effective for closed-book question answering . authors of this study found that SSM alone improves performance on temporal tasks . |
| Approach: | They introduce Temporal Span Masking (TSM) to improve performance on temporal tasks . they find that SSM alone improves the downstream performance by +5.8 points . |
| Outcome: | The proposed approach improves performance on three temporal tasks by +5.8 points . the additional targeted spans achieved by adding the TSM task are the best . |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
DenseSSM: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) face excessive computational and memory requirements due to the commonly used Transformer architecture. |
| Approach: | They propose a method to enhance the flow of hidden information between layers in large language models by selectively integrating shallow-layer hidden states into deeper layers. |
| Outcome: | The proposed method maintains parallelizability and inference efficiency of SSMs while significantly boosting performance on public benchmarks. |
SMR: State Memory Replay for Long Sequence Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing state space models (SSMs) address non-uniform sampling, but their recursive structures impede efficient SSM computation via convolution. |
| Approach: | They propose a plug-and-play mechanism to solve the Non-Stable State problem by adjusting input sequences with early memories. |
| Outcome: | The proposed method overcomes the non-uniform sample processing problem . it can achieve Sampling Step Adaptation (SSA) by adjusting input sequences with early memories. |
Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings. |
| Approach: | They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks. |
| Outcome: | The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks. |
Bridging Robustness and Generalization Against Word Substitution Attacks in NLP via the Growth Bound Matrix Approach (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that adversarial examples can alter models' predicted sentiment due to their sensitivity to specific word choices. |
| Approach: | They propose a regularization technique to improve NLP model robustness by reducing the impact of input perturbations on model outputs. |
| Outcome: | The proposed method outperforms state-of-the-art methods in adversarial defense. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
Mamba Drafters for Speculative Decoding (2025.findings-emnlp)
Copied to clipboard
Daewon Choi, Seunghyuk Oh, Saket Dingliwal, Jihoon Tack, Kyuyoung Kim, Woomin Song, Seojin Kim, Insu Han, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, Sravan Babu Bodapati
| Challenge: | Existing drafters that use external drafters suffer from slower drafting while self-speculation methods use drafters tailored to the target model but require re-training. |
| Approach: | They propose a drafter based on a state space model, Mamba, as a solution that combines the best aspects of both approaches. |
| Outcome: | The proposed drafters outperform existing drafters while using less memory and maintaining their cross-model adaptability. |
SafetyMem: Adaptive Jailbreak Defense via Dual-Component Safety Memory (2026.acl-long)
Copied to clipboard
| Challenge: | Existing defenses for Large Language Models suffer from a 'memory gap' parameter-modifying methods are computationally expensive and inference-time filters cannot retain or reuse defense knowledge across interactions. |
| Approach: | They propose a framework that secures Large Language Models through a dual-component safety memory system. |
| Outcome: | The proposed framework significantly reduces attack success rates while preserving interpretability and efficiency. |
Do Transformers Grok Succinct Algorithms? Mechanistic Evidence for Counting Circuits (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies suggest that Transformers are inherently succinct, capable of representing recursive algorithms like binary counting over exponential state spaces. |
| Approach: | They propose to bridge this gap by testing the Succinctness Hypothesis using mechanistic interpretability on a large-scale computation task. |
| Outcome: | The proposed model can represent recursive algorithms over exponential state spaces . the proposed model is able to generalize perfectly, whereas massive LSTM baselines fail completely. |