Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
Similar Papers
Answer-focused and Position-aware Neural Question Generation (D18-1)
Copied to clipboard
| Challenge: | Recent neural network-based approaches generate interrogative words that do not match the answer type. |
| Approach: | They propose an answer-focused and position-aware neural question generation model to address these issues. |
| Outcome: | The proposed model outperforms the baseline and outperformed the state-of-the-art system. |
How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures (P18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has been replaced by convolutional or self-attentional approaches. |
| Approach: | They propose an architecture definition language that allows for a flexible combination of common building blocks. |
| Outcome: | The proposed architectures can bring recurrent and convolutional models close to the Transformer architecture, but not using self-attention. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
A Recurrent BERT-based Model for Question Generation (D19-58)
Copied to clipboard
| Challenge: | Existing QG models rely on recurrent neural networks (RNNs) but the inherent sequential nature of the RNN models suffers from the problem of handling long sequences. |
| Approach: | They propose to employ a pre-trained BERT language model to tackle question generation tasks. |
| Outcome: | The proposed model outperforms the existing models on the question-answering dataset SQuAD and advances the BLEU 4 score from 16.85 to 22.17. |
Adaptive Attention Span in Transformers (P19-1)
Copied to clipboard
| Challenge: | We extend the maximum context size of a neural network called Transformer to 8k characters. |
| Approach: | They propose a self-attention mechanism that can learn its optimal attention span . this allows for models with longer context and the capability to catch longer dependencies. |
| Outcome: | The proposed model achieves state-of-the-art performance on text8 and enwiki8 using 8k characters with no loss of performance, and maintains control over memory footprint and computational time. |
Contextualized Word Representations for Reading Comprehension (N18-2)
Copied to clipboard
| Challenge: | Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content. |
| Approach: | They propose to provide a standard neural network for reading a document and answering a question about its content. |
| Outcome: | The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations. |
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer (2022.acl-long)
Copied to clipboard
| Challenge: | Existing sparse attention methods use fixed patterns to select words without considering similarities between words. |
| Approach: | They propose a neural clustering method which integrates into the Self-Attention Mechanism in Transformer and integrates it into the target task. |
| Outcome: | The proposed method outperforms two typical sparse attention methods on translation, text classification, and text matching tasks while having a comparable or even better time and memory efficiency. |
Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional transformer-based models falter due to noise sensitivity and lack explainability . ATTUN is a transformer architecture designed to enhance model transparency and resilience to noise. |
| Approach: | They propose a transformer architecture that enhances model transparency and resilience to noise . ATTUN is a module that directly modifies attention weights . they validated their approach using fact-checking datasets based on their results . |
| Outcome: | The proposed model improves predictions and identify relevant sections of input data. |
Guiding Attention for Self-Supervised Learning with Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that self-attention patterns in trained models contain a majority of non-linguistic regularities. |
| Approach: | They propose a technique to allow efficient self-supervised learning with bi-directional Transformers by using an auxiliary loss function to guide attention heads to conform to such patterns. |
| Outcome: | The proposed method achieves state-of-the-art in low-resource settings and is agnostic to pre-training objectives. |