When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks where large-scale parallel corpora cannot be obtained. |
| Approach: | They perform five sets of experiments to analyze when pre-trained word embeddings can be useful in NMT tasks. |
| Outcome: | The embeddings provide gains of up to 20 BLEU points in the most favorable setting. |
Similar Papers
Multimodal Machine Translation with Embedding Prediction (N19-3)
Copied to clipboard
| Challenge: | Pretrained word embeddings improve multimodal machine translation of low-resource domains due to a shortage of training data. |
| Approach: | They propose to combine pretrained word embeddings with search-based approaches to improve NMT of low-resource domains to better translate rare words. |
| Outcome: | The proposed approach improves translation performance by 1.24 METEOR and 2.49 BLEU and achieves 7.67 F-score. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) often fails in one-to-many translation, e.g., in the translation of multi-word expressions, compounds, and collocations. |
| Approach: | They propose a phrase-based NMT model that generates embeddings of words or phrases. |
| Outcome: | The proposed model performs on par with state-of-the-art phrase-based NMT. |
When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation? (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-training for many-to-many neural machine translation use manual cleaning of bilingual dictionaries, which are unavailable for most language pairs. |
| Approach: | They propose a word-level contrastive objective to leverage word alignments for many-to-many neural machine translation (NMT) Empirical results show that this leads to 0.8 BLEU gains for several language pairs. |
| Outcome: | Empirical results show that the proposed objective leads to 0.8 BLEU gains for several language pairs. |
Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation? (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pre-training masked language models with artificial data has been proven beneficial for several natural language processing tasks, however, it has been less explored for neural machine translation (NMT). |
| Approach: | They pre-trained masked language models with random sequences and created artificial data mimicking token frequency information from the real world. |
| Outcome: | The results show that pre-training models with artificial data improves translation performance in low-resource situations. |
Pre-training Methods for Neural Machine Translation (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. |
| Approach: | This tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. |
| Outcome: | This tutorial explains how to make the most of pre-training for neural machine translation. |
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained Transformers has focused on learning the meaning of positions . Embedding the position information in the self-attention mechanism is also an indispensable factor in NLP . |
| Approach: | They propose to use feature-level analysis to examine pre-trained Transformers' position embeddings . they also use empirical experiments to determine the appropriate positional encoding function . |
| Outcome: | The results of the empirical study can guide future work to choose the appropriate positional encoding function for specific tasks. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Revisiting Low-Resource Neural Machine Translation: A Case Study (P19-1)
Copied to clipboard
| Challenge: | Recent research has shown that neural machine translation models are highly data-inefficient and underperform phrase-based statistical machine translation (PBSMT) in low-resource settings. |
| Approach: | They propose to use auxiliary data to train low-resource neural machine translation systems without auxiliary monolingual or multilingual data. |
| Outcome: | The proposed methods outperform PBSMT and other statistical machine translation models in Korean–English with minimal data. |
On the use of BERT for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |