Learning Hard Retrieval Decoder Attention for Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | In this paper, we show that learning a hard retrieval attention that attends to a single token in a sentence is 1.43 times faster than the standard scaled dot-product attention. |
| Approach: | They propose a method to learn hard retrieval attention where an attention head attends to a single token in a sentence rather than all tokens. |
| Outcome: | The proposed method is 1.43 times faster in decoding while preserving translation quality on a wide range of MT tasks. |
Similar Papers
Hard-Coded Gaussian Attention for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has questioned the importance of multi-headed attention in achieving high translation quality. |
| Approach: | They develop a “hard-coded” attention variant without any learned parameters. |
| Outcome: | The proposed model reduces BLEU scores by adding a single learned cross attention head to an otherwise hard-coded Transformer. |
Attention over Heads: A Multi-Hop Attention for Neural Machine Translation (P19-2)
Copied to clipboard
| Challenge: | Existing multihop attentions for machine comprehension are recurrent and hierarchical . a proposed multi-hop attention for the Transformer refines the attention for an output symbol many times . |
| Approach: | They propose a multi-hop attention for the Transformer which integrates attentions from each head. |
| Outcome: | The proposed model outperforms the baseline Transformer in terms of translation accuracy and speed. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a series of experiments show that fine-tuning only the cross-attention parameters is nearly as effective as fine-timing all parameters. |
| Approach: | They conduct experiments to fine-tune a translation model on data where either the source or target language has changed. |
| Outcome: | The proposed model can be trained to several new languages with reduced parameter storage overhead. |
Synchronous Syntactic Attention for Transformer Neural Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing syntaxbased NMT models use monolingual syntactic information on either side or both. |
| Approach: | They propose a mechanism that synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target- and target side self- attentions mapped by the encoder-decoder attention matrix. |
| Outcome: | The proposed method improves translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU). |
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that attention heads learn simple positional patterns . |
| Approach: | They propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require external knowledge. |
| Outcome: | The proposed model improves translation quality and improves BLEU scores by up to 3 points in low-resource scenarios. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features. |
| Approach: | They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads. |
| Outcome: | The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters. |
Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss. |
| Approach: | They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions. |
| Outcome: | The proposed model outperforms baseline models by 0.7 BLEU in the speech task. |