Recursive Routing Networks: Learning to Compose Modules for Language Understanding (N19-1)
Copied to clipboard
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, Lauri Karttunen
| Challenge: | Recursive Routing Networks are modular, adaptable models that learn effectively in diverse environments. |
| Approach: | They propose to apply Recursive Routing Networks (RRNs) to natural language understanding by integrating them into existing architectures and recurrent network hidden layers. |
| Outcome: | The proposed model optimizes the parameters of the functions and the meta-learner decision-making component for routing inputs through those functions. |
Similar Papers
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to decompose complex tasks into simpler ones do not require annotated decompositions. |
| Approach: | They propose a framework for building interpretable systems that learn to solve complex tasks by decomposing existing models into simpler ones solvable by existing models. |
| Outcome: | The proposed framework is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Fast and Scalable Expansion of Natural Language Understanding Functionality for Intelligent Agents (N18-3)
Copied to clipboard
| Challenge: | a recent paper describes efficient deep neural network architectures for expanding natural language capabilities of virtual agents. |
| Approach: | They propose deep neural network architectures that maximize re-use available resources . they use data from Amazon Alexa to accelerate expansion of new natural language domains . |
| Outcome: | The proposed methods increase accuracy in low resource settings and enable rapid development with less data. |
A Survey on Dynamic Neural Networks for Natural Language Processing (2023.findings-eacl)
Copied to clipboard
| Challenge: | Dynamic neural networks can scale up pretrainable models with sub-linear increases in computation and time. |
| Approach: | They summarize the progress of three types of dynamic neural networks in NLP . skimming, mixtures of experts, and early exit are among the most popular . |
| Outcome: | The proposed models can scale up with sub-linear increases in computation and time . skimming, mixture of experts, and early exit are the most popular approaches . |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades. |
| Approach: | They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs. |
| Outcome: | The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic. |
Deep Bayesian Natural Language Processing (P19-4)
Copied to clipboard
| Challenge: | Introduction to deep Bayesian learning for natural language addresses the fundamentals of statistical models and neural networks. |
| Approach: | This tutorial addresses the advances in deep Bayesian learning for natural language . it focuses on advanced Bayessian models and deep models . authors present case studies and domain applications to tackle different issues . |
| Outcome: | This tutorial focuses on advanced Bayesian models and deep models for natural language . case studies and domain applications are presented to tackle different issues in deep Bayessian processing, learning and understanding. |
Information Flow Routes: Automatically Interpreting Language Models at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current state-of-the-art language models (LMs) are built on top of the Transformer architecture. |
| Approach: | They propose to build graphs where nodes correspond to token representations and edges to computations . they show that attention heads and subword merging heads are important . |
| Outcome: | The proposed model can analyze behavior for specific types of predictions, or different domains. |
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |
Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training. |
| Approach: | They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model. |
| Outcome: | The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement) |