| Challenge: | Recent studies show that recurrent networks suffer from severe computational inefficiency due to weak parallelization. |
| Approach: | They propose a lightweight recurrent network (LRN) that uses input and forget gates to handle long-range dependencies and gradient vanishing and explosion. |
| Outcome: | The proposed recurrent network yields the best running efficiency on six NLP tasks. |
Similar Papers
lrnnx: A library for Linear RNNs (2026.eacl-srw)
Copied to clipboard
Karan Bania, Soham Kalburgi, Manit Tanwar, null Dhruthi, Aditya Nagarsekar, Harshvardhan Mestha, Naman Chibber, Raj Deshmukh, Anish Sathyanarayanan, Aarush Rathore, Pratham Chheda
| Challenge: | Existing implementations of linear recurrent neural networks are fragmented across different software frameworks . existing implementations often require custom CUDA kernels or lack publicly available code altogether . |
| Approach: | a unified software library implements several modern LRNN architectures under a common interface. lrnnx aims to improve accessibility, reproducibility, and extensibility of LRnn research and applications. |
| Outcome: | lrnnx aims to improve accessibility, reproducibility, and extensibility of LRNN research and applications. |
Recurrent Attention Networks for Long-text Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to encoding long documents using self-attention have been limited by quadratic computational complexities and limited application in long text processing. |
| Approach: | They propose a long-document encoding model that allows the recurrent operation of self-attention. |
| Outcome: | The proposed model extracts global semantics in token-level and document-level representations, making it inherently compatible with both sequential and sequential tasks. |
Efficient Sequence Learning with Group Recurrent Networks (N18-1)
Copied to clipboard
| Challenge: | Recurrent neural networks have achieved state-of-the-art results in many artificial intelligence tasks, such as language modeling, neural machine translation and speech recognition. |
| Approach: | They propose an efficient architecture to improve the efficiency of such RNN model training by adopting the group strategy for recurrent layers while exploiting the representation rearrangement strategy between layers as well as time steps. |
| Outcome: | The proposed architecture achieves comparable or better accuracy compared with baselines, with a much smaller number of parameters and at a lower computational cost. |
Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)
Copied to clipboard
| Challenge: | LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. |
| Approach: | They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input. |
| Outcome: | The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients. |
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades. |
| Approach: | They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs. |
| Outcome: | The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic. |
Simple Recurrent Units for Highly Parallelizable Recurrence (D18-1)
Copied to clipboard
| Challenge: | recurrent neural networks scale poorly due to the intrinsic difficulty in parallelizing their state computations. |
| Approach: | They propose a simple recurrent unit that provides expressive recurrence and allows highly parallel implementation. |
| Outcome: | The proposed model achieves 5—9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets and delivers stronger results than LS and convolutional models. |
Reproducing and Regularizing the SCRN Model (C18-1)
Copied to clipboard
| Challenge: | Recurrent neural networks (RNNs) have demonstrated tremendous success in sequence modeling . naive dropout, variational dropout and weight tying are common techniques used to regularize the SCRN model . |
| Approach: | They propose a Structurally Constrained Recurrent Network (SCRN) model and regularize it using existing techniques. |
| Outcome: | The proposed model outperforms the LSTM model on non-English data while being much simpler. |
Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks (D18-1)
Copied to clipboard
| Challenge: | Existing gated recurrent networks have a vanishing gradient, allowing for more matrix transformations and less transparent functions. |
| Approach: | They propose an additionsubtraction twin-gated recurrent network (ATR) to simplify neural machine translation. |
| Outcome: | The proposed system is more transparent than LSTM/GRU due to the simplification. |
On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)
Copied to clipboard
| Challenge: | Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time. |
| Approach: | They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior . |
| Outcome: | The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU. |
Seq2SeqPy: A Lightweight and Customizable Toolkit for Neural Sequence-to-Sequence Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Neural models have attracted a lot of attention in the past few years due to their complexity and need to be customized to meet specific needs. |
| Approach: | They propose a lightweight toolkit for sequence-to-sequence modeling that prioritizes simplicity and ability to customize the standard architectures easily. |
| Outcome: | The proposed tool performs similarly or even better than a very widely used sequence-to-sequence toolkit. |