| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |
Similar Papers
BERTology for Machine Translation: What BERT Knows about Linguistic Difficulties for Translation (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained transformer-based models have shown excellent performance in most benchmark tests, but lack a good understanding of the linguistic knowledge of BERT in Neural Machine Translation (NMT). |
| Approach: | They propose to use QE models to analyze BERT's syntactic dependencies and their impact on machine translation quality. |
| Outcome: | The proposed model is able to model with self-attention in the pre-training phase, which improves generalization ability. |
A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | BERT is a promising technique to improve NMT, but how it outperforms standard NMT is understudied. |
| Approach: | We compare MT engines trained with pre-trained BERT and back-translation with incrementally larger amounts of data. |
| Outcome: | The proposed technique outperforms standard NMT models on morphology and syntax. |
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance. |
| Approach: | They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings. |
| Outcome: | The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model. |
Recycling a Pre-trained BERT Encoder for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | In monolingual tasks, the number of unlearned model parameters is as huge as the number learned parameters in the BERT model. |
| Approach: | They propose to apply a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to Transformer-based neural machine translation (NMT) based on the Transformer. |
| Outcome: | The proposed model is stable and efficient in low-resource settings. |
On the differences between BERT and MT encoder spaces and how to address them in translation tasks (2021.acl-srw)
Copied to clipboard
| Challenge: | Various studies show that pretrained language models cannot replace encoders in neural machine translation despite their success in other tasks. |
| Approach: | They propose a supervised transformation from one into the other to improve the applicability of BERT in neural machine translation. |
| Outcome: | The proposed transformations show that they cannot replace encoders in MT despite their success in other tasks. |
Better Neural Machine Translation by Extracting Linguistic Information from BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Experimental results show that incorporating linguistic information into neural machine translation models is no more difficult to train than conventional Transformer-based NMT. |
| Approach: | They propose to extract linguistic information from contextual word embeddings instead of point estimates to augment NMT models. |
| Outcome: | The proposed method generalizes better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
BERTGen: Multi-task Generation through BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU). |
| Approach: | They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks. |
| Outcome: | The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts. |
Pre-training Methods for Neural Machine Translation (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. |
| Approach: | This tutorial provides a comprehensive guide to make the most of pre-training for neural machine translation. |
| Outcome: | This tutorial explains how to make the most of pre-training for neural machine translation. |
Document-Level Neural Machine Translation Using BERT as Context Encoder (2020.aacl-srw)
Copied to clipboard
| Challenge: | Large-scale pre-trained representations such as BERT have been widely used in many natural language understanding tasks. |
| Approach: | They propose to use BERT as a context encoder to achieve document-level contextual information, which is then integrated into both the encoder and decoder. |
| Outcome: | The proposed model outperforms strong document-level machine translation baselines on BLEU score and captures document- level context information to boost translation performance. |