RNNS

Deep RNNs Encode Soft Hierarchical Syntax (P18-2)

Challenge:	Recurrent neural tensor networks (RNNs) increase capacity by augmenting the size of the hidden layer, with significant increase in computational cost.
Approach:	They propose restricted recurrent neural tensor networks (r-RNTNs) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights .
Outcome:	The proposed model outperforms unrestricted RNTNs using only a small fraction of the parameters of unrestrained RNNNs.

Copied to clipboard

Terra Blevins, Omer Levy, Luke Zettlemoyer

Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks (2023.eacl-srw)

Challenge:	Existing studies show that syntactic information is useful for a wide variety of NLP tasks.
Approach:	They propose to use word-level representations to learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision.
Outcome:	The proposed model encodes significant amounts of syntax even without explicit supervision.

Copied to clipboard

Nadine El-Naggar, Pranava Madhyastha, Tillman Weyde

Challenge:	Existing studies have shown that linear RNNs with unbounded activation functions are difficult to train effectively and do not learn exact counting behaviour.
Approach:	They propose to identify the necessary conditions for a linear single-cell RNN to have the ability to count and to investigate how these conditions relate to the empirical behaviour of trained linear RNN models.
Outcome:	The proposed model is a linear single-cell RNN with an unbounded activation function and a Dyck-1-like balanced bracket language.

The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)

Copied to clipboard

Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao

Joey NMT: A Minimalist NMT Toolkit for Novices (D19-3)

Challenge:	MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models .
Approach:	They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop.
Outcome:	The proposed model can significantly compress a large model without significant performance drop.

Copied to clipboard

Julia Kreutzer, Jasmijn Bastings, Stefan Riezler

Multilinguality

Challenge:	a recent study shows that novices perform better than experts in a code quiz.
Approach:	They present a minimalist neural machine translation toolkit based on PyTorch . they evaluate the accessibility of the toolkit in a user study .
Outcome:	The proposed toolkit performs comparable to more complex toolkits on standard benchmarks.

Multimodal Modeling of Task-Mediated Confusion (2022.naacl-srw)

Copied to clipboard

Camille Mince, Skye Rhomberg, Cecilia Alm, Reynold Bailey, Alex Ororbia

Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)

Challenge:	Existing models for confusion detection are under-explored, but they can be used to detect it computationally.
Approach:	They build upon prior work to develop models that detect confusion from three modalities: video, audio, and text.
Outcome:	The proposed models can detect confusion from facial expressions, prosody, and transcribed spoken language.

Copied to clipboard

Hongfei Xu, Josef van Genabith, Deyi Xiong, Qiuhui Liu, Jingyi Zhang

A Formal Hierarchy of RNN Architectures (2020.acl-main)

Challenge:	Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies .
Approach:	They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships.
Outcome:	The proposed model can be computed in parallel and improves on the WMT 14 tasks.

Copied to clipboard

William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, Eran Yahav

Challenge:	Existing theories of expressive power of RNNs are limited.
Approach:	They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence.
Outcome:	The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models.

Linear-time Constituency Parsing with RNNs and Dynamic Programming (P18-2)

Copied to clipboard

Juneki Hong, Liang Huang

Randomized Deep Structured Prediction for Discourse-Level Processing (2021.eacl-main)

Challenge:	Existing span-based constituency parsers are too slow for longer sentences and for applications beyond sentence boundaries.
Approach:	They propose a linear-time constituency parser with RNNs and dynamic programming using graph-structured stack and beam search.
Outcome:	The proposed parser is faster for long sentences and faster for discourse parsing.

Copied to clipboard

Manuel Widmoser, Maria Leonor Pacheco, Jean Honorio, Dan Goldwasser

Colorless Green Recurrent Networks Dream Hierarchically (N18-1)

Challenge:	Expressive text encoders have been at the center of recent NLP work . however, some tasks require complex structural dependencies between texts .
Approach:	They propose to leverage deep structured prediction and expressive neural encoders for argumentation mining tasks.
Outcome:	The proposed framework can be used for argumentation mining tasks without expensive inference tools.

Copied to clipboard

Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni

Convolutional Neural Networks with Recurrent Neural Filters (D18-1)

Challenge:	Recurrent neural networks (RNNs) can induce non-trivial properties of language.
Approach:	They investigate whether RNNs can track hierarchical syntactic structure . they include nonsensical sentences where RNN cannot rely on semantic cues .
Outcome:	The proposed models can predict long-distance agreement in nonsensical sentences in Italian and English.

Copied to clipboard

Yi Yang

Challenge:	Convolutional neural networks (CNNs) use recurrent neural networks as convolution filters to capture language compositionality and long-term dependencies.
Approach:	They propose to use recurrent neural networks (RNNs) as convolution filters to capture language compositionality and long-term dependencies.
Outcome:	The proposed convolutional neural networks achieve state-of-the-art on two sentences and the Stanford Sentiment Treebank.

Linguistic Knowledge and Transferability of Contextual Representations (N19-1)

Copied to clipboard

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith

Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)

Challenge:	Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence.
Approach:	They compare pretrained word representations with 16 diverse probing tasks to examine their transferability.
Outcome:	The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds .

Copied to clipboard

Omer Levy, Kenton Lee, Nicholas FitzGerald, Luke Zettlemoyer

Implicit n-grams Induced by Recurrence (2022.naacl-main)

Challenge:	LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections.
Approach:	They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input.
Outcome:	The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

Copied to clipboard

Xiaobing Sun, Wei Lu

Responsible & Trustworthy NLP

Challenge:	Recent studies show that self-attention based models have limitations on modeling sequential transformations.
Approach:	They propose to extract some explainable features from trained RNNs that are reminiscent of classical n-grams features.
Outcome:	The proposed models can model interesting linguistic phenomena such as negation and intensification.

On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)

Copied to clipboard

Gail Weiss, Yoav Goldberg, Eran Yahav

Fast and Accurate Reordering with ITG Transition RNN (C18-1)

Challenge:	Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time.
Approach:	They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior .
Outcome:	The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU.

Copied to clipboard

Hao Zhang, Axel Ng, Richard Sproat

Machine Translation

Challenge:	Attention-based sequence-to-sequence neural networks learn to jointly align and translate.
Approach:	They propose to use a reordering RNN that shares the input encoder with the decoder to decouple re-ordering from translation.
Outcome:	The proposed model can achieve superior reordering accuracy without feature engineering and is 2.5x faster in decoding.

On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages (2020.coling-main)

Copied to clipboard

Satwik Bhattamishra, Kabir Ahuja, Navin Goyal

Challenge:	recurrent models have been effective in NLP tasks but performance on context-free languages (CFLs) is weak.
Approach:	They evaluate the performance of recurrent models on Dyck-n languages . they find that they are expressive enough to recognize Dyck words of arbitrary lengths if their depths are bounded.
Outcome:	The proposed models generalize well on Dyck-n languages, while performing poorly on longer test strings.

Language is All a Graph Needs (2024.findings-eacl)

Copied to clipboard

Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang

Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons (D19-1)

Challenge:	Existing work on integrating graph problems into generative language modeling framework remains limited.
Approach:	They propose an LLM with instructions based on natural language to perform graph tasks.
Outcome:	The proposed model surpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets and sheds light on generative LLMs as new foundation model for graph machine learning.

Copied to clipboard

Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, Zhaopeng Tu

Learning to Write with Cooperative Discriminators (P18-1)

Challenge:	Recent studies have shown that a hybrid of self-attention networks (SANs) and recurrent neural networks (RNNs) outperforms both individual architectures, while not much is known about why the hybrid models work.
Approach:	They propose to use an advanced variant of self-attention networks (SANs) to enhance the strength of hybrid models by introducing a syntax-oriented inductive bias to perform tree-like composition.
Outcome:	The proposed model outperforms both individual models and a standard hybrid model on a machine translation task.

Copied to clipboard

Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, Yejin Choi

Text Generation

Challenge:	Despite their local fluency, long-form text generated from RNNs is often generic, repetitive, and even self-contradictory.
Approach:	They propose a unified learning framework that can guide a base RNN generator towards more globally coherent generations by combining discriminators with a composite decoding objective.
Outcome:	The proposed framework can guide a base RNN generator towards more globally coherent generations by combining discriminators with the base RRN generator through a composite decoding objective.

RNNs can generate bounded hierarchical languages with optimal memory (2020.emnlp-main)

Copied to clipboard

John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, Christopher D. Manning

Global-Local Modeling with Prompt-Based Knowledge Enhancement for Emotion Inference in Conversation (2023.findings-eacl)

Challenge:	Existing studies have shown that RNNs can efficiently generate bounded hierarchical languages with high syntactic fidelity, but their success is not well-understood theoretically.
Approach:	They propose a language of well-nested brackets and m-bounded nesting depth . they prove that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction.
Outcome:	The proposed language is well-nested brackets and has m-bounded nesting depth . it shows that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction.

Copied to clipboard

Renxi Wang, Shi Feng

PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism (2023.acl-long)

Challenge:	Existing studies on emotion recognition focus on recognizing emotions through a speaker’s utterance, while research on emotion inference predicts emotions of addressees through previous utterations.
Approach:	They propose a global-local modeling method based on recurrent neural networks and pre-trained language models to do emotion inference in conversation.
Outcome:	The proposed method achieves state-of-the-art on three datasets.

Copied to clipboard

Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze

Dialogue Systems & Conversational Agents

Challenge:	Existing generative models for dialogue use the last hidden state to summarize the history of the dialogue.
Approach:	They propose a Pseudo-Variational Gated Recurrent Unit (PVGRU) that summarises the accumulated distribution variations of subsequences and builds a model based on it.
Outcome:	The proposed model can improve diversity and relevance of responses on two benchmark datasets.

Representations and Architectures in Neural Sentiment Analysis for Morphologically Rich Languages: A Case Study from Modern Hebrew (C18-1)

Copied to clipboard

Adam Amram, Anat Ben David, Reut Tsarfaty

To Attend or not to Attend: A Case Study on Syntactic Structures for Semantic Relatedness (P18-1)

Challenge:	Existing sentiment analyzers for MRLs that use tokens and morpheme-based representations have no empirically studied effects of representation choices on neural sentiment analysis.
Approach:	They develop a sentiment analysis benchmark for Hebrew based on 12K social media comments and provide two instances of data.
Outcome:	The proposed benchmarks show that representation choices have measurable effects on task perfromance and that they vary depending on architecture type.

Copied to clipboard

Amulya Gupta, Zhu Zhang

ModRWKV: Transformer Multimodality in Linear Time (2025.emnlp-main)

Challenge:	Recent success of Recurrent Neural Networks (RNNs) in Machine Translation (MT) has prompted attention mechanisms to be used in machine translation.
Approach:	They propose a tree-structured attention model on Tree Long Short-Term Memory Networks . they also experiment with three LSTM variants: bidirectional-LSTMs, Constituency Tree-LSTS, and Dependency Tree LSTS.
Outcome:	The proposed model is based on tree-LSTMs, constituency trees, and dependencies trees.

Copied to clipboard

Jiale Kang, Ziyin Yue, Qingyu Yin, Rui Jiang, Weile Li, Zening Lu, Zhouran Ji

Recurrent Neural Networks as Weighted Language Recognizers (N18-1)

Challenge:	Currently, multimodal studies are based on large language models with quadratic-complexity Transformer architectures.
Approach:	They propose a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone and a lightweight architecture to achieve multi-source information fusion.
Outcome:	The proposed framework achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders.

Copied to clipboard

Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, Kevin Knight

Challenge:	Recent experiments show that RNNs outperform other methods in assigning high probability to held-out English text.
Approach:	They focus on the single-layer, ReLU-activation, rational-weight RNNs with softmax . they show that most problems for such RNN are undecidable .
Outcome:	The proposed model outperforms other methods in assigning high probability to held-out English text.

Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention (2021.naacl-main)

Copied to clipboard

Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, Jacob Andreas

GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (P19-1)

Challenge:	Existing approaches to compositional generalization in semantic parsers focus on word-level alignments, but they focus on spans.
Approach:	They propose a span-level supervised attention loss that improves compositional generalization in semantic parsers by focusing on spans.
Outcome:	The proposed method improves on three benchmarks of compositional generalization.

Copied to clipboard

Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, Jie Zhou

Chunking

Challenge:	Existing systems for sequence labeling are limited by shallow connections between consecutive hidden states and insufficient modeling of global information.
Approach:	They propose a global context enhanced deep transition architecture for sequence labeling . they deepen the state transition path at each position in a sentence and assign tokens with global representations .
Outcome:	The proposed architecture outperforms the best reported results on two standard sequence labeling tasks.

On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)

Copied to clipboard

Anej Svete, Robin Chan, Ryan Cotterell

Sliced Recurrent Neural Networks (C18-1)

Challenge:	Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs).
Approach:	They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed.
Outcome:	The results suggest that RNNs can represent a larger class of LMs than previously claimed .

Copied to clipboard

Zeping Yu, Gongshen Liu

Sentiment Analysis

Challenge:	Recurrent neural networks have difficulty in parallelization because of their recurrent structure.
Approach:	They propose sliced recurrent neural networks (SRNNs) which can be parallelized by slicing sequences into many subsequences.
Outcome:	The proposed recurrent neural networks perform better than standard RNNs on six large-scale sentiment analysis datasets.

On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing (N19-1)

Copied to clipboard

Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, Nanyun Peng

Cross-Lingual Transfer

Challenge:	Existing studies on crosslingual transfer have focused on word-level information sharing, but words are not independent in sentences; their combinations form larger linguistic units, known as context.
Approach:	They propose to use orderagnostic models to transfer word order to distant languages . they train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages.
Outcome:	The proposed model performs better on languages with different word orders than on other languages.

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization (2022.emnlp-main)

Copied to clipboard

Zhiyuan Zhang, Ruixuan Luo, Qi Su, Xu Sun

TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model (2023.findings-acl)

Challenge:	Recent studies show flat minima tend to imply better generalization abilities . however, it has some difficulty implying SAM to some natural language tasks .
Approach:	They propose a flatness-aware minimization algorithm that can be applied to natural language tasks . they propose to use parameter corruptions to explain why flat minima generalize better .
Outcome:	The proposed algorithm can generalize better for flat minima that are robust against corruptions or perturbations.

Copied to clipboard

Patrick Kahardipraja, Brielen Madureira, David Schlangen

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (P19-1)

Challenge:	Recent approaches for incremental processing use RNNs or Transformers, which consume whole sequences and are by nature non-incremental.
Approach:	They propose a two-pass model for AdaPtIve Revision to obtain an incremental supervision signal for learning an adaptive revision policy.
Outcome:	The proposed model has better incremental performance and faster inference speed compared to restart-incremental Transformers while showing little degradation on full sequences.

Copied to clipboard

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, Ruslan Salakhutdinov

An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models (2020.acl-main)

Challenge:	Term memory networks (RNNs) are difficult to optimize due to gradient vanishing and explosion.
Approach:	They propose a neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence.
Outcome:	The proposed method improves state-of-the-art performance on short and long sequences and generates coherent, novel text articles with thousands of tokens.

Copied to clipboard

Hiroshi Noji, Hiroya Takamura

Bayesian Compression for Natural Language Processing (D18-1)

Challenge:	Neural language models are often trained on positive examples, but recent studies suggest they are not robust enough to handle complex syntactic constructions.
Approach:	They propose to use negative examples to boost models' robustness on English sentences with a negligible loss of perplexity.
Outcome:	The proposed model is robust to negative examples in English with negligible loss of perplexity .

Copied to clipboard

Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov

Explainability & Interpretability in NLP

Challenge:	In natural language processing, recurrent neural networks have a huge number of parameters.
Approach:	They propose a Bayesian sparsification technique which allows compressing RNNs dozens or hundreds of times without time-consuming hyperparameters tuning.
Outcome:	The proposed technique compresses the RNN dozens or hundreds of times without time-consuming hyperparameters tuning.

On the Relationship Between RNN Hidden-State Vectors and Semantic Structures (2024.findings-acl)

Copied to clipboard

Edi Muskardin, Martin Tappler, Ingo Pill, Bernhard Aichernig, Thomas Pock

Text Clustering

Challenge:	Using hidden-state vectors of recurrent neural networks (RNNs) we examine the assumption that hidden- state vectors tend to form clusters of semantically similar vectors, which we dub the clustering hypothesis.
Approach:	They propose to use recurrent neural networks (RNNs) that model processes with internal states to test their hypothesis.
Outcome:	The proposed model is based on a set of RNNs that were trained to recognize regular languages and a context-free language.

Porous Lattice Transformer Encoder for Chinese NER (2020.coling-main)

Copied to clipboard

Xue Mengge, Bowen Yu, Tingwen Liu, Yue Zhang, Erli Meng, Bin Wang

Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)

Challenge:	Existing methods to integrate word boundary information into character-level Chinese NER are inefficient and lack semantic interaction.
Approach:	They propose an extension of transformer encoder that is tailored for ChineseNER to incorporate lexicons into character-level Chinese NER by lattices.
Outcome:	The proposed extension performs 11.4 times faster than state-of-the-art methods while retaining the rich long-term dependencies.

Copied to clipboard

Machel Reid, Edison Marrese-Taylor, Yutaka Matsuo

Continuous Language Generative Flow (2021.acl-long)

Challenge:	Recent improvements in NLP tasks can be attributed to the Transformer model.
Approach:	They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization.
Outcome:	The proposed model outperforms the current RNN model even with significantly fewer parameters.

Copied to clipboard

Zineng Tang, Shiyue Zhang, Hyounghun Kim, Mohit Bansal

Text Generation

Challenge:	Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformers.
Approach:	They propose a flow-based language generation model that adapts flow-derived generative models to language generation via continuous input embeddings, adapted affine coupling structures, and a novel architecture for autoregressive text generation.
Outcome:	The proposed model improves on QG and NMT and improves performance over baselines on SQuAD and TVQA and NML16.

Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)

Copied to clipboard

Shauli Ravfogel, Yoav Goldberg, Tal Linzen

Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training (D19-1)

Challenge:	Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks.
Approach:	They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features .
Outcome:	The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias.

Copied to clipboard

Alham Fikri Aji, Kenneth Heafield, Nikolay Bogoychev

Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)

Challenge:	In recent years, neural network models have grown dramatically in terms of number of parameters, so exchanging gradients during data-parallel training is costly in terms both of bandwidth and time.
Approach:	They propose to combine the compressed global gradient with the local gradient to restore Transformer convergence while RNNs converge faster.
Outcome:	The proposed method restores transformer convergence while RNNs converge faster.

Copied to clipboard

Anej Svete, Franz Nowak, Anisha Sahabdeen, Ryan Cotterell

CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks (P19-1)

Challenge:	Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages.
Approach:	They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity.
Outcome:	The proposed models can express arbitrary regular LMs with linearly bounded precision.

Copied to clipboard

Roberto Dessì, Marco Baroni

Attention-based Conditioning Methods for External Knowledge Integration (P19-1)

Challenge:	Recent deep neural network successes rekindled debates on their natural language processing abilities.
Approach:	They propose to test the ability of sequence-to-sequence networks to perform systematic, compositional generalization of linguistic rules.
Outcome:	The proposed dataset shows that convolutional networks perform better on compositional generalization tasks than RNNs.

Copied to clipboard

Katerina Margatina, Christos Baziotis, Alexandros Potamianos

Semi-supervised URL Segmentation with Recurrent Neural Networks Pre-trained on Knowledge Graph Entities (2020.coling-main)

Challenge:	Existing approaches for incorporating external knowledge into deep neural networks (RNNs) lexicon features are used to concatenate external information into the input or hidden network layers.
Approach:	They propose a method for conditioning external knowledge into RNNs by concatenating a representation of the external information to the input or hidden network layers.
Outcome:	The proposed approach improves performance on six benchmark datasets.

Copied to clipboard

Hao Zhang, Jae Ro, Richard Sproat

On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)

Challenge:	Domain names such as openresearch are being added to a growing set of tokens that an NLP system may need to deal with.
Approach:	They propose a tagging model that uses characters as input to break domain names into component words . they propose taagging methods that use concatenated entity names in a large knowledge database .
Outcome:	The proposed model improves on concatenated entity names in a knowledge database by 33% . the proposed model can be used for a wide range of languages, including Chinese and Japanese .

Copied to clipboard

Franz Nowak, Anej Svete, Li Du, Ryan Cotterell

Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (D18-1)

Challenge:	Existing studies have focused on LMs as formal languages, but they do not consider language membership.
Approach:	They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine .
Outcome:	The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings .

Copied to clipboard

Gongbo Tang, Mathias Müller, Annette Rios, Rico Sennrich

Corpora for Document-Level Neural Machine Translation (2020.lrec-1)

Challenge:	Recent studies show that non-recurrent architectures outperform RNNs in neural machine translation.
Approach:	They hypothesize that CNNs and self-attentional networks could extract semantic features from source text.
Outcome:	The proposed architectures outperform RNNs on two tasks: subject-verb agreement and word sense disambiguation.

Copied to clipboard

Siyou Liu, Xiaojun Zhang

Multilinguality

Challenge:	Document-level machine translation models translate sentences in isolation, but there are three main problems for document-level models.
Approach:	They propose to use document-level machine translation to capture discourse dependencies across sentences by considering a document as a whole.
Outcome:	The proposed method captures discourse dependencies across sentences by considering a document as a whole.

Sound Signal Processing with Seq2Tree Network (L18-1)

Copied to clipboard

Weicheng Ma, Kai Cao, Zhaoheng Ni, Peter Chin, Xiang Li

Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)

Challenge:	Recent LSTM models have been used to model sequential data processing tasks because of their ability to preserve previous information weighted on distance.
Approach:	They propose to use a tree-structured tree-based neural network architecture to solve the problem of unbalanced connections between data units inside and outside semantic groups.
Outcome:	The proposed model outperforms the state-of-the-art Bidirectional LSTM model on a signal and noise separation task.

Copied to clipboard

Anej Svete, Ryan Cotterell

The Importance of Being Recurrent for Modeling Hierarchical Structure (D18-1)

Challenge:	Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages.
Approach:	They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N \|\|right) neurons to represent an LM .
Outcome:	The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models.

Copied to clipboard

Ke Tran, Arianna Bisazza, Christof Monz

Challenge:	Recent work shows that recurrent neural networks can implicitly capture hierarchical information when trained to solve common natural language processing tasks.
Approach:	They propose a convolutional sequence-to-sequence model that exploits hierarchical information implicitly.
Outcome:	The proposed model is recurrent and non-recurrent, and it can model hierarchical structure implicitly.

Pointing to Select: A Fast Pointer-LSTM for Long Text Classification (2020.coling-main)

Copied to clipboard

Jinhua Du, Yan Huang, Karo Moilanen

Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)

Challenge:	Existing methods to skip irrelevant words in text processing are slow and vanishing gradients can cause slow inference and a loss of coherence.
Approach:	They propose a pointer network-based LSTM framework which can change skip rates during inference.
Outcome:	The proposed model is 1.1x3.5x faster than the standard LSTM framework and more accurate than Leap-LSTM at high skip rates.

Copied to clipboard

Hung Le, Doyen Sahoo, Nancy Chen, Steven Hoi

Scaling up the State Size of RNN LLMs for Long-Context Scenarios (2025.acl-long)

Challenge:	Existing work on video-grounded dialogue systems is limited by feature space and semantic information.
Approach:	They propose multimodal transformer networks to encode videos and incorporate information from different modalities.
Outcome:	The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance.

Copied to clipboard

Kai Liu, Jianfei Gao, Kai Chen

Language Modeling with a General Second-Order RNN (2020.lrec-1)

Challenge:	Existing RNN-based LLMs struggle with long-context scenarios due to their quadratic computational complexity and linear memory requirements.
Approach:	They propose an efficient scaling method to scale RNN models to match the 2k context length of Transformers with small parameters overhead.
Outcome:	The proposed method improves long-context understanding and improves performance on FDA recall-intensive tasks.

Copied to clipboard

Diego Maupomé, Marie-Jean Meurs

Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)

Challenge:	a number of RNNs update their state as the input sequence is processed . second-order RNN architectures show promising performance in language modeling .
Approach:	They propose a second-order RNN architecture that generalizes existing ones . they use a Penn Treebank dataset to analyze how their different components affect performance .
Outcome:	The proposed architecture generalizes existing RNNs on a Penn Treebank dataset . it shows that removing the first-order terms does not hinder performance .

Copied to clipboard

Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber

Embedding Dynamic Attributed Networks by Modeling the Evolution Processes (2020.coling-main)

Challenge:	Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions.
Approach:	They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed .
Outcome:	The proposed extensions overcome limitations of the LT and self-referential weight matrices.

Copied to clipboard

Zenan Xu, Zijing Ou, Qinliang Su, Jianxing Yu, Xiaojun Quan, ZhenKun Lin

Representation Learning

Challenge:	Existing methods to embed nodes into low-dimensional vectors focus on static networks, but in practice, many networks are evolving over time and hence are dynamic, e.g., social networks.
Approach:	They propose to extract high-order neighborhood information at each given timestamp and then use an embedding prediction framework to capture the temporal correlations.
Outcome:	Extensive experiments on four real-world datasets show that the proposed method outperforms baseline methods for dynamic link prediction and node classification tasks.

Transformer Based Multi-Source Domain Adaptation (2020.emnlp-main)

Copied to clipboard

Dustin Wright, Isabelle Augenstein

Responsible & Trustworthy NLP

Challenge:	Existing approaches to improve machine learning performance are mixed experts and domain adversarial training.
Approach:	They investigate the problem of unsupervised multi-source domain adaptation . they combine predictions of multiple domain experts and combine them to induce a domain agnostic representation space .
Outcome:	The proposed methods improve models' performance while limiting learning time.

Automatic Labeling of Problem-Solving Dialogues for Computational Microgenetic Learning Analytics (L18-1)

Copied to clipboard

Yuanliang Meng, Anna Rumshisky, Florence Sullivan

Dialogue Systems & Conversational Agents

Challenge:	This paper presents a recurrent neural network model to automate the analysis of students' computational thinking in problem-solving dialogue.
Approach:	They propose a recurrent neural network model to automate the analysis of students' computational thinking in problem-solving dialogue.
Outcome:	The proposed model outperforms the baseline model and outperformed the nave model by a large margin.

On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning (2024.acl-long)

Copied to clipboard

Franz Nowak, Anej Svete, Alexandra Butoi, Ryan Cotterell

VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models (2025.coling-main)

Challenge:	Existing theoretical treatments have shed light on some aspects of CoT, but we are still far from a concrete understanding of the concept.
Approach:	They propose to formalize CoT reasoning in a probabilistic setting to bridge this gap . they show that LMs can represent the same family of distributions over strings as probabilistic Turing machines.
Outcome:	The proposed model can represent the same family of distributions over strings as Turing machines.

Copied to clipboard

Haowen Hou, Peigen Zeng, Fei Ma, Fei Richard Yu

Finetuning Pretrained Transformers into RNNs (2021.emnlp-main)

Challenge:	Visual Language Models (VLMs) have been gaining popularity with large language models, but few attempts have been made to incorporate efficient linear Recurrent Neural Networks (RNNs) into VLMs.
Approach:	They propose a linear RNN model with a data-dependent recurrence and sandwich prompts to enhance modeling capabilities and a 2D image scanning mechanism to enrich the processing of visual sequences.
Outcome:	The proposed model achieves competitive performance compared to Transformer-based models on various benchmarks.

Copied to clipboard

Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith

RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)

Challenge:	Efficient transformers outperform recurrent neural networks in natural language generation, but this comes with significant computational cost and memory footprint during generation.
Approach:	They propose to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy.
Outcome:	The proposed transformers outperform recurrent neural networks in natural language generation but come with significant computational and memory footprint during generation.

Copied to clipboard

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu

Transformers are Multi-State RNNs (2024.emnlp-main)

Challenge:	recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability.
Approach:	They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs.
Outcome:	The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models.

Copied to clipboard

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, Roy Schwartz