Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Similar Papers
Synchronous Syntactic Attention for Transformer Neural Machine Translation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing syntaxbased NMT models use monolingual syntactic information on either side or both. |
| Approach: | They propose a mechanism that synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target- and target side self- attentions mapped by the encoder-decoder attention matrix. |
| Outcome: | The proposed method improves translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU). |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Syntax-guided Localized Self-attention by Constituency Syntactic Distance (2022.findings-emnlp)
Copied to clipboard
Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, Zhouhan Lin
| Challenge: | Recent studies have shown that Transformers is implicitly learning syntactic information from data, albeit is highly dependent on the quality and scale of the training data. |
| Approach: | They propose a syntax-guided localized self-attention model that allows directly incorporating grammar structures from an external constituency parser. |
| Outcome: | The proposed model improves translation performance on a variety of datasets, from small to large datasets and with different source languages. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features. |
| Approach: | They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads. |
| Outcome: | The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters. |
Self-Attention with Relative Position Representations (N18-2)
Copied to clipboard
| Challenge: | Recent approaches to sequence to sequence learning leverage recurrence, convolution, attention or combination of recurrent and convolutional neural networks. |
| Approach: | They propose an approach that extends the self-attention mechanism to consider representations of relative positions, or distances between sequence elements. |
| Outcome: | The proposed approach yields 1.3 BLEU and 0.3 BLUE on translation tasks . it is based on a relation-aware self-attention mechanism that can generalize to arbitrary graph-labeled inputs. |
Training Deeper Neural Machine Translation Models with Transparent Attention (D18-1)
Copied to clipboard
| Challenge: | Existing NMT models are shallow in comparison to convolutional models used for both text and vision tasks. |
| Approach: | They propose to modify the attention mechanism to ease the optimization of deeper models by a simple modification to the seq2seq with attention paradigm. |
| Outcome: | The proposed model achieves consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT'15 Czech-English tasks. |
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a series of experiments show that fine-tuning only the cross-attention parameters is nearly as effective as fine-timing all parameters. |
| Approach: | They conduct experiments to fine-tune a translation model on data where either the source or target language has changed. |
| Outcome: | The proposed model can be trained to several new languages with reduced parameter storage overhead. |
Multilingual Neural Machine Translation with Task-Specific Attention (C18-1)
Copied to clipboard
| Challenge: | Multilingual machine translation is a task of building a system capable of translating between multiple source and target languages. |
| Approach: | They propose task-specific attention models to retain parameter sharing generalization . they observe improved translation quality even in low-resource zero-shot directions . |
| Outcome: | The proposed model retains parameter sharing generalization while allowing language-specific specialization . it improves translation quality even in low-resource zero-shot translation directions . |
Multi-Granularity Self-Attention for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing neural machine translation models use a deep multi-head self-attention network with no explicit phrase information. |
| Approach: | They propose a neural network that combines multi-head self-attention and phrase modeling to train attention heads to attend to phrases in either n-gram or syntactic formalisms. |
| Outcome: | The proposed approach improves on English-to-German and NIST Chinese-to English translation tasks. |