| Challenge: | Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. |
| Approach: | They propose to use Monte Carlo dropout and deep ensembles to quantify uncertainty in machine translation and assess their ability to target different sources of aleatoric and epistemic uncertainty. |
| Outcome: | The proposed measures can target different sources of aleatoric and epistemic uncertainty, with a reduction in computational costs. |
Similar Papers
Uncertainty-Aware Machine Translation Evaluation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Several neural-based metrics have been proposed to evaluate machine translation quality, but they are trained on noisy, biased and scarce human judgements. |
| Approach: | They propose a method to evaluate machine translation quality using point estimates . they combine COMET framework with Monte Carlo dropout and deep ensembles . |
| Outcome: | The proposed methods perform well across multiple language pairs and with references. |
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization (2023.acl-long)
Copied to clipboard
| Challenge: | evaluators of machine translation systems often use text-based metrics to evaluate performance . however, these metrics lack semantic-level information and exhibit poor correlation with human ratings . authors propose a method to reduce inference bias of neural metrics in out-of-distribution data . |
| Approach: | They propose to reduce inference bias by using uncertainty estimation, test-time adaptation, and inference to reduce model uncertainty. |
| Outcome: | The proposed method reduces model uncertainty and improves correlation performance across models. |
Uncertainty Quantification for Evaluating Gender Bias in Machine Translation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models can reproduce existing social inequalities but cannot be reduced. |
| Approach: | They propose that models should maintain uncertainty when input is ambiguous to avoid reinforcing biases. |
| Outcome: | The proposed model can detect gender bias when translated to ambiguous and unambiguous sources and shows that it does not correlate with high translation accuracy and debiases the two cases differently. |
COMET: A Neural Framework for MT Evaluation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Historically, metrics for evaluating the quality of machine translation (MT) have relied on basic, lexical-level features such as counting the number of matching n-grams between the MT hypothesis and the reference translation. |
| Approach: | They propose a neural framework for training multilingual machine translation evaluation models which exploits human judgements to obtain new state-of-the-art levels of correlation with MT quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems. |
Multi-Hypothesis Machine Translation Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. |
| Approach: | They propose to use MT models to generate multiple diverse translations and use them as surrogates to reference translations to obtain a quantification of translation variability. |
| Outcome: | The proposed approach improves correlation with human judgements of quality by 15%. |
Difficulty-Aware Machine Translation Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Current MT evaluation measures pay the same attention to each sentence component . in real-world examinations, the questions vary in difficulty and weightings . |
| Approach: | They propose a difficulty-aware MT evaluation metric that takes translation difficulty into account . they propose to use this metric to evaluate machine translation (MT) results . |
| Outcome: | The proposed method outperforms most MT evaluation metrics in terms of human correlation. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |