Confidence-Aware Scheduled Sampling for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation models are trained to maximize the likelihood of the next token given previous golden tokens as inputs, but at the inference stage, golden token is unavailable. |
| Approach: | They propose a scheduled sampling method that randomly replaces groundtruth tokens with predicted ones during training, ignoring real-time model competence. |
| Outcome: | The proposed method outperforms the Transformer and vanilla scheduled sampling on large-scale translations. |
Similar Papers
Scheduled Sampling Based on Decoding Steps for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation models are trained to maximize the likelihood of next token given previous golden tokens as inputs, but at the inference stage, golden token is unavailable. |
| Approach: | They propose to use scheduled sampling to replace ground-truth tokens with predicted tokens to bridge the gap between training and inference. |
| Outcome: | The proposed methods outperform the Transformer baseline and vanilla scheduled sampling on three large-scale WMT tasks. |
Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between ground-truth prefixes used during training and model-generated prefix at inference time. |
| Approach: | They propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance. |
| Outcome: | The proposed method significantly outperforms maximum likelihood estimation and scheduled sampling baselines on four translation datasets. |
Scheduled Sampling for Transformers (P19-2)
Copied to clipboard
| Challenge: | Existing studies show that scheduled sampling can be applied to recurrent neural networks to avoid exposure bias. |
| Approach: | They propose to use teacher forced embeddings and model predictions to avoid exposure bias in sequence-to-sequence generation. |
| Outcome: | The proposed technique achieves performance close to a teacher-forcing baseline on two language pairs and is promising for future research. |
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to correct exposure bias in machine translation are inadequate . scheduled sampling assumes that words are aligned at each time step . |
| Approach: | They propose a differentiable sampling algorithm that optimizes the probability that the reference can be aligned with the sampled output. |
| Outcome: | The proposed approach improves BLEU on translation tasks and is simpler to train with no sampling schedule. |
Towards Robust Neural Machine Translation with Iterative Scheduled Data-Switch Training (2022.coling-1)
Copied to clipboard
Zhongjian Miao, Xiang Li, Liyan Kang, Wen Zhang, Chulun Zhou, Yidong Chen, Bin Wang, Min Zhang, Jinsong Su
| Challenge: | Existing methods on robust neural machine translation (NMT) construct adversarial examples by injecting noise into authentic examples and indiscriminately exploit two types of examples. |
| Approach: | They propose an iterative scheduled data-switch training framework to mitigate this problem by injecting noise into authentic examples and indiscriminately exploiting two types of examples. |
| Outcome: | The proposed model outperforms several competitive benchmarks on four translation benchmarks. |
Dynamic Oracle for Neural Machine Translation in Decoding Phase (L18-1)
Copied to clipboard
| Challenge: | Existing methods to improve NMT performance but there is a discrepancy between training and inference when decoding. |
| Approach: | They propose to use Scheduled Sampling to reduce the discrepancy between training and inference in NMT when decoding to mitigate the discrépancy. |
| Outcome: | The proposed methods improve translation quality over standard NMT system. |
Bridging the Gap between Training and Inference for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural Machine Translation generates target words sequentially while at inference it has to generate the entire sequence from scratch. |
| Approach: | They propose to use ground truth and inference to generate target words sequentially while at inference it has to generate the entire sequence from scratch. |
| Outcome: | Experiments on Chinese->English and WMT’14 English->German translation tasks show that the proposed model can achieve significant improvements on multiple datasets. |
Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that enhancing the learning on uncertain monolingual sentences improves the translation quality of high-uncertainty sentences and also benefits the prediction of low-frequency words at the target side. |
| Approach: | They propose to use monolingual data to augment model training with synthetic parallel data by selecting the most informative monolingual sentences to complement the parallel data. |
| Outcome: | The proposed approach improves the performance of natural language models by selecting the most informative monolingual sentences. |
Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Existing studies show that training on a single related language is more effective than using all data. |
| Approach: | They propose an efficient algorithm that first samples a target sentence, and then conditionally samples its source sentence. |
| Outcome: | The proposed algorithm brings significant gains on three of four languages with minimal training overhead. |
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization (2023.acl-long)
Copied to clipboard
| Challenge: | evaluators of machine translation systems often use text-based metrics to evaluate performance . however, these metrics lack semantic-level information and exhibit poor correlation with human ratings . authors propose a method to reduce inference bias of neural metrics in out-of-distribution data . |
| Approach: | They propose to reduce inference bias by using uncertainty estimation, test-time adaptation, and inference to reduce model uncertainty. |
| Outcome: | The proposed method reduces model uncertainty and improves correlation performance across models. |