Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy (2024.eacl-long)
Copied to clipboard
| Challenge: | Uncertainty estimation is an important diagnostic tool for statistical models. |
| Approach: | They propose to adapt similarity-sensitive Shannon entropy (S3E) for NMT by incorporating a concept borrowed from theoretical ecology. |
| Outcome: | The proposed framework improves quality estimation and named entity recall, and improves translation quality. |
Similar Papers
Uncertainty-Aware Semantic Augmentation for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for neural machine translation only observe one source sentence at training time . this discrepancy in data distribution leads to a formidable learning challenge . |
| Approach: | They propose an uncertainty-aware semantic augmentation approach to capture universal semantic information among multiple source sentences and enhance hidden representations with this information. |
| Outcome: | The proposed approach outperforms baseline and existing methods on translation tasks. |
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate long one-sentence responses that are less effective because they overlook two crucial factors: intra-cluster similarity and inter-c cluster similarity. |
| Approach: | They propose a method that generalizes semantic entropy and uses token probabilities to quantify uncertainty in large language models. |
| Outcome: | The proposed method can be extended to white-box settings by incorporating token probabilities. |
Uncertainty-Aware Curriculum Learning for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) has proven to be facilitated by curriculum learning which presents examples in an easy-to-hard order at different training stages. |
| Approach: | They propose to use an uncertainty-aware curriculum learning approach to assess data difficulty and model competence to provide examples in an easy-to-hard order at different training stages. |
| Outcome: | The proposed approach outperforms baseline and related methods on translation quality and convergence speed. |
Improving Back-Translation with Uncertainty-based Confidence Estimation (D19-1)
Copied to clipboard
| Challenge: | Despite the success of low-resource neural machine translation, there is a data scarcity problem in many languages . large-scale, high-quality, and widecoverage bilingual corpora do not exist for most language pairs . |
| Approach: | They propose to quantify confidence of NMT models based on model uncertainty . they propose to use uncertainty-based confidence measures to improve back-translation . |
| Outcome: | The proposed model outperforms conventional statistical machine translation (SMT) on Chinese-English and English-German translation tasks. |
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Prior work treats all types of mismatches between source and target as noise . Consequently, it remains unclear how noisy parallel training samples impact NMT training. |
| Approach: | They propose a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences. |
| Outcome: | The proposed framework improves translation quality and model calibration on EN-FR tasks. |
Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation models are often criticized for failures that happen without competency awareness. |
| Approach: | They propose a method that extends conventional NMT with a self-estimator to translate a source sentence and estimate its competency. |
| Outcome: | The proposed method performs on translation tasks intact and on quality estimation tasks better than existing methods. |
Quality-Aware Decoding for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, Andre Martins
| Challenge: | Despite advances in machine translation quality estimation and evaluation, decoding is mostly oblivious to this. |
| Approach: | They propose to use a decoding framework that is quality-aware for neural machine translation . they compare various methods like N-best reranking and minimum Bayes risk decoding . |
| Outcome: | The proposed quality-aware decoding outperforms MAP-based decoding on four datasets and two model classes. |
Handling Homographs in Neural Machine Translation (N18-1)
Copied to clipboard
| Challenge: | Existing methods for MT have problems with translating homographs, as it is difficult to select the correct translation based on the context. |
| Approach: | They propose to model the context of the input word with context-aware word embeddings that help to differentiate the word sense before feeding it into the encoder. |
| Outcome: | The proposed models improve translation accuracy and BLEU score on three language pairs. |
Uncertainty-Aware Machine Translation Evaluation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Several neural-based metrics have been proposed to evaluate machine translation quality, but they are trained on noisy, biased and scarce human judgements. |
| Approach: | They propose a method to evaluate machine translation quality using point estimates . they combine COMET framework with Monte Carlo dropout and deep ensembles . |
| Outcome: | The proposed methods perform well across multiple language pairs and with references. |
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization (2023.acl-long)
Copied to clipboard
| Challenge: | evaluators of machine translation systems often use text-based metrics to evaluate performance . however, these metrics lack semantic-level information and exhibit poor correlation with human ratings . authors propose a method to reduce inference bias of neural metrics in out-of-distribution data . |
| Approach: | They propose to reduce inference bias by using uncertainty estimation, test-time adaptation, and inference to reduce model uncertainty. |
| Outcome: | The proposed method reduces model uncertainty and improves correlation performance across models. |