Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for unsupervised neural machine translation (UNMT) use cross-lingual pretraining to align the lexical- and high-level representations of two languages. |
| Approach: | They propose to use type-level cross-lingual subword embeddings to enhance the bilingual masked language model pretraining with lexical-level information to align the two languages. |
| Outcome: | Empirical results show that the method improves on UNMT (up to 4.5 BLEU) and bilingual lexicon induction compared to baseline models. |
Similar Papers
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) models with limited data are ineffective when the two languages are not available for one language. |
| Approach: | They propose an approach that reuses a language model that is pretrained on two languages with large monolingual data to initialize an unsupervised neural machine translation system. |
| Outcome: | The proposed method outperforms a competitive cross-lingual pretraining model in English-Macedonian (En-Mk) and English-Albanian (En Sq) it yields more than +8.3 BLEU points for all four translation directions. |
Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation methods have been observed to make particular errors in comparison to supervised machine translation, such as confusing nouns that pertain to the same semantic category. |
| Approach: | They propose a method that incorporates images at the word level to augment lexical mappings. |
| Outcome: | Experiments on a multi-lingual dataset show that the proposed method generates more accurate translations with only monolingual data. |
Unsupervised Bilingual Word Embedding Agreement for Unsupervised Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Unsupervised bilingual word embedding (UBWE) has helped unsupervised neural machine translation (UNMT) achieve remarkable results in several language pairs. |
| Approach: | They propose two methods that train UNMT with UBWE agreement . they propose to use UBwe to initialize word embedding in UNMT . |
| Outcome: | The proposed methods outperform conventional methods on several language pairs. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found that the performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text in a target language. |
| Approach: | They propose to use bilingual lexicons to synthesize textual or labeled data and combine it with monolingual or parallel text when available. |
| Outcome: | The proposed methods improve performance for 19 under-represented languages with and without extra monolingual text. |
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to build initial unsupervised machine translation models with cross-lingual n-gram embeddings are inexplicit and limited. |
| Approach: | They propose a cross-lingual pre-training method that incorporates cross-linguistic training signals into pre-trained models by randomly choosing source n-grams in the input text stream. |
| Outcome: | The proposed method significantly improves the performance of unsupervised machine translation. |
Exploring Unsupervised Pretraining Objectives for Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised cross-lingual pretraining has significantly reduced the need for large parallel data. |
| Approach: | They compare unsupervised cross-lingual pretraining with masking and reconstructing inputs in the decoder to produce real sentences. |
| Outcome: | The proposed methods produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. |
Improving Lexically Constrained Neural Machine Translation with Source-Conditioned Masked Span Prediction (2021.acl-short)
Copied to clipboard
| Challenge: | Accurate terminology translation is crucial for ensuring the practicality and reliability of neural machine translation systems. |
| Approach: | They propose a method to preserve terminology in translations as lexical constraints with or without a term dictionary at test time. |
| Outcome: | The proposed setup achieves consistent improvements on terminology and sentence-level translation for three domain-specific corpora in two language pairs. |
Advances and Challenges in Unsupervised Neural Machine Translation (2021.eacl-tutorials)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation (UNMT) has achieved impressive results, but there are still several challenges for the technology. |
| Approach: | They present a framework for unsupervised neural machine translation (UNMT) they examine the latest progress and challenges of UNMT and examine how it holds up . |
| Outcome: | The proposed method has achieved impressive results but still faces challenges. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |