| Challenge: | Neural machine translation models are vulnerable to unfamiliar inputs. |
| Approach: | They propose to drop tokens of the input sentences to improve generalization and avoid overfitting for the NMT model. |
| Outcome: | The proposed approach improves on Chinese-English and English-Romanian benchmarks and achieves significant performance improvements over baselines. |
Similar Papers
Token-level Adaptive Training for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural machine translation (NMT). |
| Approach: | They propose to assign tokens with different frequencies to target tokens during training to encourage the model to pay more attention to low-frequency tokens. |
| Outcome: | The proposed model yields consistent improvements on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens. |
A Simple and Effective Approach to Coverage-Aware Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models are used to solve translation problems using long-term models. |
| Approach: | They propose a method to seek a better balance between model confidence and length preference for Neural Machine Translation. |
| Outcome: | The proposed model improves on Chinese-English and English-German translation tasks. |
The Effects of Language Token Prefixing for Multilingual Machine Translation (2022.aacl-short)
Copied to clipboard
| Challenge: | In recent years, the field has moved towards large neural models either translating from or into many languages. |
| Approach: | They propose to prefix language tokens onto a source or target sequence to improve translation performance. |
| Outcome: | The proposed methods improve translation performance and source side prefixes improve translation. |
Token-wise Curriculum Learning for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of “easy” samples from training data at the early stage of training. |
| Approach: | They propose a token-wise curriculum learning approach that creates sufficient amounts of easy samples from training data. |
| Outcome: | The proposed approach outperforms baselines on five language pairs on low-resource languages. |
Revisiting Negation in Neural Machine Translation (2021.tacl-1)
Copied to clipboard
| Challenge: | Negation is an important linguistic phenomenon in machine translation, as errors in translating negation may change the meaning of source sentences completely. |
| Approach: | They evaluate the translation of negation in English–German (EN–DE) and English– Chinese (EN-ZH) . they find that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation . |
| Outcome: | The accuracy of manual evaluation in ENDE, DEEN, ENZH, and ZHEN is 95.7%, 94.8%, 93.4%, and 91.7% respectively. |
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing methods for data augmentation for text-based tasks such as machine translation are limited due to noise and noise. |
| Approach: | They propose a data augmentation policy with desirable properties as an optimization problem and propose 'SwitchOut' switchout randomly replaces words in both the source and target sentences with other random words from their corresponding vocabularies. |
| Outcome: | The proposed method outperforms strong alternatives such as word dropout on three translation datasets. |
On the Sparsity of Neural Machine Translation Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Modern neural machine translation models employ a large number of parameters, which leads to serious over-parameterization. |
| Approach: | They propose to prune parameters to improve the model by +0.8 BLEU points and to reallocate them to enhance the ability of modeling low-level lexical information. |
| Outcome: | The pruned parameters improve the model by +0.8 BLEU points and the rejuvenated parameters enhance the ability to model low-level lexical information. |
Zero-shot North Korean to English Neural Machine Translation by Character Tokenization and Phoneme Decomposition (2020.acl-srw)
Copied to clipboard
| Challenge: | a limited number of North Korean to English translation models have been developed . a zero-shot approach is proposed to train a neural machine translation model using South Korean data . |
| Approach: | They propose a method to tokenize South Korean input sentences and decompose them into phonemes. |
| Outcome: | The proposed method improves the BLEU scores by +1.01 points compared with the baseline . the proposed method can learn North Korean to English translation and improve the linguistic accuracy. |
Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) relies on source sentence and target prefix attributions for each input token. |
| Approach: | They propose an interpretability method that tracks input tokens’ attributions for both contexts and extends it to any encoder-decoder Transformer-based model. |
| Outcome: | The proposed method can be extended to any encoder-decoder Transformer-based model and provides insights into their behaviour. |
Revisiting Token Dropping Strategy in Efficient BERT Pretraining (2023.acl-long)
Copied to clipboard
| Challenge: | Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT. |
| Approach: | They propose a semantic-consistent learning method to improve token dropping by skipping the computation of a subset of input tokens at several middle layers. |
| Outcome: | The proposed method achieves consistent and significant performance gains across all tasks and model sizes. |