StateX: Enhancing RNN Recall via Post-training State Expansion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that RNNs with large recurrent states are expensive to train . however, the ability to recall contextual information from long contexts is underperforms them in certain aspects. |
| Approach: | They propose a framework that expands the states of pre-trained RNNs by scaling them up to 1.3B . they use a recurrent architecture that compresses contextual information into a fixedsize state . |
| Outcome: | Experiments on models with up to 1.3B parameters show that StateX expands state sizes without incurring high post-training costs or compromising other capabilities. |
Similar Papers
Scaling up the State Size of RNN LLMs for Long-Context Scenarios (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RNN-based LLMs struggle with long-context scenarios due to their quadratic computational complexity and linear memory requirements. |
| Approach: | They propose an efficient scaling method to scale RNN models to match the 2k context length of Transformers with small parameters overhead. |
| Outcome: | The proposed method improves long-context understanding and improves performance on FDA recall-intensive tasks. |
Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings. |
| Approach: | They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks. |
| Outcome: | The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks. |
RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)
Copied to clipboard
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
| Challenge: | recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability. |
| Approach: | They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. |
| Outcome: | The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models. |
Language Modeling with a General Second-Order RNN (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of RNNs update their state as the input sequence is processed . second-order RNN architectures show promising performance in language modeling . |
| Approach: | They propose a second-order RNN architecture that generalizes existing ones . they use a Penn Treebank dataset to analyze how their different components affect performance . |
| Outcome: | The proposed architecture generalizes existing RNNs on a Penn Treebank dataset . it shows that removing the first-order terms does not hinder performance . |
A Formal Hierarchy of RNN Architectures (2020.acl-main)
Copied to clipboard
| Challenge: | Existing theories of expressive power of RNNs are limited. |
| Approach: | They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence. |
| Outcome: | The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models. |
Attention-based Conditioning Methods for External Knowledge Integration (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for incorporating external knowledge into deep neural networks (RNNs) lexicon features are used to concatenate external information into the input or hidden network layers. |
| Approach: | They propose a method for conditioning external knowledge into RNNs by concatenating a representation of the external information to the input or hidden network layers. |
| Outcome: | The proposed approach improves performance on six benchmark datasets. |
How Well Can a Long Sequence Model Model Long Sequences? Comparing Architectural Inductive Biases on Long-Context Abilities (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in system engineering and model design have enabled extended context models. |
| Approach: | They propose to scale up models that are purported to support extended contexts . they show that recurrent models still suffer in the same settings as long-context LLMs if attention is given to them . |
| Outcome: | The proposed models can extend to infinite sequence length, but they suffer in the same settings as long-context models with attention. |
Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adaptive Retention is a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. |
| Approach: | They propose a probabilistic token selection mechanism that learns which representations to keep under a strict global budget M. |
| Outcome: | The proposed method reduces memory usage by 35–45% while improving throughput by 1.8. |
On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)
Copied to clipboard
| Challenge: | Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time. |
| Approach: | They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior . |
| Outcome: | The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU. |
Beyond Fixed-Length Calibration for Post-Training Compression of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has demonstrated that the sequence length of calibration data plays a crucial role in the effectiveness of post-training compression methods. |
| Approach: | They propose a calibration technique that applies masking along the sequence axis to normalized hidden states. |
| Outcome: | The proposed method improves perplexity and zero-shot downstream tasks performance. |