Recursive Tree-Structured Self-Attention for Answer Sentence Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Recent top-performing models in Answer Sentence Selection use self-attention and transfer learning, but not syntactic structure. |
| Approach: | They propose a recursive, tree-structured self-attention model that can represent all levels of syntactic parse trees with only one additional layer. |
| Outcome: | The proposed model can represent all levels of syntactic parse trees with only one additional layer without transfer learning. |
Similar Papers
You Only Need Attention to Traverse Trees (P19-1)
Copied to clipboard
| Challenge: | Recent research has focused on sentence representations. |
| Approach: | They propose a tree-based model that captures phrase-level syntax and word-level dependencies by doing recursive traversal with attention. |
| Outcome: | a new model captures phrase-level syntax and word-level dependencies with attention. |
Tree Transformer: Integrating Tree Structures into Self-Attention (D19-1)
Copied to clipboard
| Challenge: | Existing work on hierarchical structure in neural networks has not captured human intuitions about hierarchic structures. |
| Approach: | They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder to encourage attention heads to follow tree structures. |
| Outcome: | The proposed model improves language modeling and learning more explainable attention scores. |
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
A Gated Self-attention Memory Network for Answer Selection (D19-1)
Copied to clipboard
| Challenge: | Existing deep learning approaches for answer selection use word-level comparison followed by aggregation. |
| Approach: | They propose a new gated self-attention memory network for answer selection task . they combine a transfer learning technique from a large-scale online corpus to create a gated network . |
| Outcome: | The proposed model outperforms existing methods on two standard answer selection datasets: TrecQA and WikiQA. |
To Attend or not to Attend: A Case Study on Syntactic Structures for Semantic Relatedness (P18-1)
Copied to clipboard
| Challenge: | Recent success of Recurrent Neural Networks (RNNs) in Machine Translation (MT) has prompted attention mechanisms to be used in machine translation. |
| Approach: | They propose a tree-structured attention model on Tree Long Short-Term Memory Networks . they also experiment with three LSTM variants: bidirectional-LSTMs, Constituency Tree-LSTS, and Dependency Tree LSTS. |
| Outcome: | The proposed model is based on tree-LSTMs, constituency trees, and dependencies trees. |
Composition, Attention, or Both? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests that language models implicitly learn syntactic structures of natural language, even though they do not receive explicit syntatic supervision. |
| Approach: | They propose a novel architecture that recursively compose subtrees with a composition function and selectively attend to previous structural information with sc-attention mechanisms. |
| Outcome: | The proposed architecture can induce human-like syntactic generalization by recursive composition and selective attention to previous structural information. |
Rethinking Self-Attention: Towards Interpretability in Neural Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that attention mechanisms provide arguably explainable attention distributions that can help to interpret predictions. |
| Approach: | They propose a new self-attention layer where attention heads represent labels. |
| Outcome: | The proposed model obtains state-of-the-art results on the Penn Treebank and Chinese Treebank. |
DRTS Parsing with Structure-Aware Encoding and Decoding (2020.acl-main)
Copied to clipboard
| Challenge: | Discourse representation tree structure (DRTS) parsing is a new semantic parser which ignores structural information. |
| Approach: | They propose a structural-aware model to integrate structural information into the model . they use graph attention network (GAT) to exploit structural information for effective modeling . |
| Outcome: | The proposed model can achieve the best performance on a benchmark dataset. |
Learning Sentence Representations over Tree Structures for Target-Dependent Classification (N18-1)
Copied to clipboard
| Challenge: | Existing work on tree structures uses syntactic parsers or Treebank annotations to perform target-dependent classifications. |
| Approach: | They propose a reinforcement learning based approach which automatically induces target-specific sentence representations over tree structures. |
| Outcome: | The proposed model gives superior performance on two benchmark tasks compared to previous work on parsed trees . |
Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) models are based on sequential decoding or serialisation of structured data into sequence. |
| Approach: | They propose a model that combines sequential encoder with tree-structured decoding augmented with a syntax-aware attention model. |
| Outcome: | The proposed model produces fluent translations with better reordering than previous models. |