| Challenge: | Recent research has focused on sentence representations. |
| Approach: | They propose a tree-based model that captures phrase-level syntax and word-level dependencies by doing recursive traversal with attention. |
| Outcome: | a new model captures phrase-level syntax and word-level dependencies with attention. |
Similar Papers
Tree Transformer: Integrating Tree Structures into Self-Attention (D19-1)
Copied to clipboard
| Challenge: | Existing work on hierarchical structure in neural networks has not captured human intuitions about hierarchic structures. |
| Approach: | They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder to encourage attention heads to follow tree structures. |
| Outcome: | The proposed model improves language modeling and learning more explainable attention scores. |
Recursive Tree-Structured Self-Attention for Answer Sentence Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Recent top-performing models in Answer Sentence Selection use self-attention and transfer learning, but not syntactic structure. |
| Approach: | They propose a recursive, tree-structured self-attention model that can represent all levels of syntactic parse trees with only one additional layer. |
| Outcome: | The proposed model can represent all levels of syntactic parse trees with only one additional layer without transfer learning. |
Roles and Utilization of Attention Heads in Transformer-based Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Sentence encoders based on transformer architectures have shown promising results on various natural language understanding tasks. |
| Approach: | They propose a sentence representation method that takes advantage of most influential attention heads. |
| Outcome: | The proposed method improves performance on the downstream tasks. |
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited. |
| Approach: | They propose an unsupervised method that extracts constituency trees from PLM attention heads. |
| Outcome: | The proposed method outperforms existing approaches if no development set is present. |
On Tree-Based Neural Sentence Modeling (D18-1)
Copied to clipboard
| Challenge: | Existing tree-based sentence modeling approaches adopt syntactic parsing trees as the explicit structure prior. |
| Approach: | They replace parsing trees with trivial trees to study their effectiveness . they found that tree-based sentence modeling gives better results when crucial words are closer to the final representation . |
| Outcome: | The proposed tree-based sentences have shown better results on many downstream tasks. |
Syntax-Based Attention Masking for Neural Machine Translation (2021.naacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to extend transformers to source-side trees are linearized into sequences, but they are limited by positional encodings. |
| Approach: | They propose a method for extending transformers to source-side trees by using masks based on tree positions . they define a number of masks that limit self-attention based upon relationships among tree nodes . |
| Outcome: | The proposed method improves on translations from English to germany and English to english and germany by +2.1 BLEU. |
To Attend or not to Attend: A Case Study on Syntactic Structures for Semantic Relatedness (P18-1)
Copied to clipboard
| Challenge: | Recent success of Recurrent Neural Networks (RNNs) in Machine Translation (MT) has prompted attention mechanisms to be used in machine translation. |
| Approach: | They propose a tree-structured attention model on Tree Long Short-Term Memory Networks . they also experiment with three LSTM variants: bidirectional-LSTMs, Constituency Tree-LSTS, and Dependency Tree LSTS. |
| Outcome: | The proposed model is based on tree-LSTMs, constituency trees, and dependencies trees. |
Attention Can Reflect Syntactic Structure (If You Let It) (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has attempted to decode linguistic structure from the Transformer . but, much of the work focused on English, a language with rigid word order and a lack of inflectional morphology. |
| Approach: | They propose to fine-tune a feature encoder for BERT to learn linguistic structure from its multi-head attention mechanism. |
| Outcome: | The proposed model can decode full trees above baseline accuracy from single attention heads across languages. |
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Training and inference using large transformer models can be computationally expensive because the self-attention's time and memory grow quadratically with sequence length. |
| Approach: | They propose a modified transformer architecture that constrains the encoder-decoder attention mechanism to a subset of input sentences while maintaining system performance. |
| Outcome: | The proposed architecture can be trained and inferenced using large transformer models with expensive training and induction costs. |
Phrase-level Self-Attention Networks for Universal Sentence Encoding (D18-1)
Copied to clipboard
| Challenge: | Phrase-level self-attention networks (PSAN) can capture context dependencies at the phrase level instead of the sentence level. |
| Approach: | They propose to perform self-attention across words inside a phrase to capture context dependencies at the phrase level and use the gated memory updating mechanism to refine each word’s representation hierarchically with longer-term context dependency captured in a larger phrase. |
| Outcome: | The proposed model can achieve state-of-the-art performance across a plethora of NLP tasks including binary and multi-class classification, natural language inference and sentence similarity. |