Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons (D19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that a hybrid of self-attention networks (SANs) and recurrent neural networks (RNNs) outperforms both individual architectures, while not much is known about why the hybrid models work. |
| Approach: | They propose to use an advanced variant of self-attention networks (SANs) to enhance the strength of hybrid models by introducing a syntax-oriented inductive bias to perform tree-like composition. |
| Outcome: | The proposed model outperforms both individual models and a standard hybrid model on a machine translation task. |
Similar Papers
The Importance of Being Recurrent for Modeling Hierarchical Structure (D18-1)
Copied to clipboard
| Challenge: | Recent work shows that recurrent neural networks can implicitly capture hierarchical information when trained to solve common natural language processing tasks. |
| Approach: | They propose a convolutional sequence-to-sequence model that exploits hierarchical information implicitly. |
| Outcome: | The proposed model is recurrent and non-recurrent, and it can model hierarchical structure implicitly. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (D18-1)
Copied to clipboard
| Challenge: | Recent studies show that non-recurrent architectures outperform RNNs in neural machine translation. |
| Approach: | They hypothesize that CNNs and self-attentional networks could extract semantic features from source text. |
| Outcome: | The proposed architectures outperform RNNs on two tasks: subject-verb agreement and word sense disambiguation. |
Self-Attention with Structural Position Representations (D19-1)
Copied to clipboard
| Challenge: | Experimental results show that SANs can't encode positions of input words . SAN's are currently lacking in encoding positions of words based on position-unaware "bagof-words" theory . |
| Approach: | They propose to augment SANs with structural position representations to capture latent structure of input sentence. |
| Outcome: | The proposed approach consistently outperforms the sequential representations on translation tasks. |
Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing (2022.lrec-1)
Copied to clipboard
| Challenge: | A variety of hierarchical RNN models have been proposed to incorporate hierarchically-based hierarchic information in modeling languages in the literature. |
| Approach: | They propose a latent indicator layer approach to identify and learn hierarchical information and develop an EM algorithm to handle the latent indicators layer in training. |
| Outcome: | The proposed approach outperforms other RNN-based models in document classification tasks. |
How Does Selective Mechanism Improve Self-Attention Networks? (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that selective SANs outperform the standard SAN by paying more attention to content words that contribute to the meaning of the sentence. |
| Approach: | They propose to implement selective SANs with a flexible Gumbel-Softmax to improve word order encoding and structure modeling. |
| Outcome: | The proposed system outperforms the standard SANs on several representative NLP tasks including natural language inference, semantic role labelling, and machine translation. |
Deep Attentive Sentence Ordering Network (D18-1)
Copied to clipboard
| Challenge: | Existing methods for sentence ordering tasks rely on linguistic knowledge and are domain specific. |
| Approach: | They propose a deep attentive sentence ordering network which integrates self-attention mechanism with LSTMs in the encoding of input sentences. |
| Outcome: | The proposed model outperforms the state-of-the-art models on Sentence Ordering and Order Discrimination tasks and is shown to be highly efficient. |
Assessing the Ability of Self-Attention Networks to Learn Word Order (P19-1)
Copied to clipboard
| Challenge: | Existing studies have attributed SAN to being weak at learning positional information for sequence modeling due to lack of recurrence structure. |
| Approach: | They propose a word reordering detection task to quantify how well word order information is learned by SAN and RNN. |
| Outcome: | The proposed task quantifies how well word order information learned by SAN and RNN is learned. |
Self-Attention Networks Can Process Bounded Hierarchical Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that can process formal languages with hierarchical structure are limited in their performance. |
| Approach: | They propose to use a subset of Dyck-k with depth bounded by D to train self-attention networks. |
| Outcome: | The proposed model can process Dyck-(k, D) with depth bounded by D, which better captures the hierarchical structure of natural language. |
Obligation and Prohibition Extraction Using Hierarchical RNNs (P18-2)
Copied to clipboard
| Challenge: | Existing methods for contract element extraction and contract element classification focus on indicative tokens, but they are not as efficient as the current ones. |
| Approach: | They propose a self-attention mechanism that converts each sentence to an embedding and processes the embeddables to classify each sentence. |
| Outcome: | The proposed method outperforms the flat BILSTM classifier even when it considers surrounding sentences because it has a broader discourse view. |