| Challenge: | a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations. |
| Approach: | They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective . |
| Outcome: | The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average. |
Similar Papers
MR-P: A Parallel Decoding Algorithm for Iterative Refinement Non-Autoregressive Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Non-autoregressive neural machine translation models remove dependency between tokens in the target sentence and generate all tokens on parallel . |
| Approach: | They propose a non-autoregressive neural machine translation model that decodes with the Mask-Predict algorithm which iteratively refines the output. |
| Outcome: | The proposed algorithm increases the performance of the WMT’14 translation task by 1.39 points. |
Inference Strategies for Machine Translation with Conditional Masking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Conditional masked language model training has proven successful for non-autoregressive and semi-auto-regressively sequence generation tasks. |
| Approach: | They propose a conditional masked language model (CMLM) that is a factorization of conditional probabilities of partial sequences and propose heuristics to improve performance. |
| Outcome: | The proposed algorithm is more efficient than the standard “mask-predict” algorithm on machine translation tasks. |
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | masked language models have been used for natural language processing tasks but few studies have adopted it in the sequence-to-sequence models. |
| Approach: | They propose to combine encoder and decoder to train a masked sequence-to-sequence model . they propose to train the encoder more rigorously by masking the encoded input . |
| Outcome: | The proposed model achieves 27.69/32.24 BLEU scores on English-German/German-English tasks with 5+ times speed up compared with an autoregressive model. |
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism (2020.coling-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive models generate target words in parallel, but with a large latency due to the left-to-right dependency. |
| Approach: | They propose to train a conditional masked translation model and refine results within several iterations to remedy a flawed translation by non-autoregressive models. |
| Outcome: | The proposed model outperforms state-of-the-art models by over 1 BLEU while using less training computations. |
Universal Conditional Masked Language Pre-training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT) this paper demonstrates that pre-training a sequence- to-squence model with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non-autoregressive NMT tasks. |
| Approach: | They propose a conditional masked language model pre-trained on bilingual and monolingual corpora in many languages. |
| Outcome: | The proposed model can achieve significant performance improvements on all scenarios from low- to extremely high-resource languages. |
Incorporating a Local Translation Mechanism into Non-autoregressive Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to capture local dependencies among output tokens are not efficient, causing errors of repeated translation. |
| Approach: | They propose a local autoregressive translation mechanism that predicts a short sequence of tokens for each target decoding position instead of one token. |
| Outcome: | Empirical results show that the proposed method achieves comparable or better performance with fewer decoding iterations, bringing a 2.5x speedup. |
Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable results on several NLP tasks. |
| Approach: | They propose three new masking strategies for cross-lingual visual pre-training that focus on learning different linguistic patterns. |
| Outcome: | The proposed methods outperform the baseline model and achieve state-of-the-art accuracy on the Portuguese-English MMT task. |
Accelerating Transformer Inference for Translation via Parallel Decoding (2023.acl-long)
Copied to clipboard
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, Emanuele Rodola
| Challenge: | Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT) Existing methods to solve this problem are expensive and require changes to the model. |
| Approach: | They propose to reframe autoregressive decoding with a parallel formulation . they propose to speed up existing models without training or modifications while retaining translation quality. |
| Outcome: | The proposed model speeds up existing models without training or modifications while retaining translation quality. |
Masked Language Model Scoring (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained masked language models require finetuning for most tasks. |
| Approach: | They evaluate pretrained masked language models out of the box via their pseudo-log-likelihood scores (PLLs) they attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 . |
| Outcome: | The proposed model outperforms autoregressive language models in a variety of tasks. |
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train language models on diverse text corpora have brought up performance improvements on several natural language understanding (NLU) tasks. |
| Approach: | They propose a method to automatically generate domain- and task-adaptive maskings of a given text for self-supervised pre-training. |
| Outcome: | The proposed framework outperforms rule-based masking strategies on question answering and text classification datasets on which it outperformed rule-driven masking techniques. |