| Challenge: | Neural machine translation with source-side attention has been criticized for its poor memory performance. |
| Approach: | They propose to use a Decoding History Enhanced Attention mechanism to render NMT models better at selecting both source-side and target-side information. |
| Outcome: | The proposed model improves by 0:9 BLEU on Chinese-English translation and the state-of-the-art on a larger task. |
Similar Papers
Neural Hidden Markov Model for Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Attention-based neural machine translation models selectively focus on specific source positions to produce a translation. |
| Approach: | They propose to replace the attention component with a neural hidden Markov model that selectively focuss on specific source positions to produce a translation. |
| Outcome: | The proposed model performs better than the state-of-the-art attention-based models on the GermanEnglish and ChineseEnglish translation tasks. |
Look Harder: A Neural Machine Translation Model with Hard Attention (P19-1)
Copied to clipboard
| Challenge: | Soft-attention based Neural Machine Translation models attend all the words in the source sequence for each target token, which makes them ineffective for long sequence translation. |
| Approach: | They propose a hard-attention based NMT model which selects a subset of source tokens for each target token to effectively handle long sequence translation. |
| Outcome: | The proposed model performs better on long sequences and achieves significant improvement on English-German and English-French translation tasks compared to soft-attention based models. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Using attention weights, we show that NMT models make alignment errors by relying on uninformative tokens from the source sequence. |
| Approach: | They propose to use attention weights to regulate alignment errors in NMT models . they propose methods that largely reduce the word alignment error rate compared to standard induced alignments from attention weighted tokens. |
| Outcome: | The proposed methods reduce the word alignment error rate compared to standard induced alignments from attention weights. |
Document-Level Neural Machine Translation with Hierarchical Attention Networks (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) can be improved by including document-level contextual information. |
| Approach: | They propose a hierarchical attention model that captures document-level contextual information and conditioning on the NMT model’s own hidden states. |
| Outcome: | The proposed model improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods and that both the encoder and decoder benefit from context in complementary ways. |
Synchronous Syntactic Attention for Transformer Neural Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing syntaxbased NMT models use monolingual syntactic information on either side or both. |
| Approach: | They propose a mechanism that synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target- and target side self- attentions mapped by the encoder-decoder attention matrix. |
| Outcome: | The proposed method improves translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU). |
Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation models treat decoding at each time step equally with the same matrix . conventional methods treat decoder outputs at all time steps with the identical weight matrix causing inaccuracy . |
| Approach: | They propose a model with a mechanism to control the softness of attention by means of an attention temperature. |
| Outcome: | The proposed model outperforms baseline models on Chinese-English and English-Vietnamese translations. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Training Deeper Neural Machine Translation Models with Transparent Attention (D18-1)
Copied to clipboard
| Challenge: | Existing NMT models are shallow in comparison to convolutional models used for both text and vision tasks. |
| Approach: | They propose to modify the attention mechanism to ease the optimization of deeper models by a simple modification to the seq2seq with attention paradigm. |
| Outcome: | The proposed model achieves consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT'15 Czech-English tasks. |
Exploiting Deep Representations for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation models typically implement encoder and decoder as multiple layers, but only the top layers are leveraged in the subsequent process, which misses the opportunity to exploit useful information embedded in other layers. |
| Approach: | They propose to expose all of these signals with layer aggregation and multi-layer attention mechanisms and introduce an auxiliary regularization term to encourage different layers to capture diverse information. |
| Outcome: | The proposed approach exposes all of these signals with layer aggregation and multi-layer attention mechanisms on widely-used translation datasets. |