Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing variants of Kendall's tau have weaknesses in handling of ties, and in some situations can even be gamed. |
| Approach: | They propose to use Kendall's tau to meta-evaluate machine translation evaluation metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties and a tie calibration procedure that automatically introduces ties into metric scores. |
| Outcome: | The proposed model gives metrics credit for correctly predicting ties and introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. |
Similar Papers
Don’t Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to calculate agreement are acceq and acc eq, but there are many ways to compute agreement based on how scores are grouped together. |
| Approach: | They propose a segment-level meta-evaluation metric that utilizes pairwise differences rather than raw scores to refine Global Pearson to intra-segment comparisons. |
| Outcome: | The proposed metric correctly ranks sentinel evaluation metrics and better aligns with human error weightings than acceq. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to metric meta-evaluation focus on general statements about absolute and relative quality of metrics across arbitrary system outputs, but in practice, metrics are applied in highly contextual settings. |
| Approach: | They propose a method for contextual metric meta-evaluation by comparing local metric accuracy. |
| Outcome: | The proposed method compares the local metric accuracy of evaluation metrics across translation, speech recognition, and ranking tasks. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024.acl-long)
Copied to clipboard
| Challenge: | a decade ago a single metric, BLEU, governed progress in machine translation research. |
| Approach: | They investigate the "dynamic range" of a number of modern machine translation metrics to provide a collective understanding of differences in scores . they use a large dataset to discover deltas at which metrics achieve system-level differences that are meaningful to humans . |
| Outcome: | The proposed method is more stable than statistical p-values in regards to testset size. |
Can Automatic Metrics Assess High-Quality Translations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent human evaluation study found that translations produced by current MT systems achieve very high-quality scores when judged by humans on a direct assessment scale of 0 to 100. |
| Approach: | They stress-test the ability of current translation quality metrics to detect correct translations . they show that current metrics often over or underestimate translation quality . |
| Outcome: | The proposed method overestimates translation quality, the authors show . they show that current metrics often overestimate translation quality . |
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation (2026.acl-long)
Copied to clipboard
| Challenge: | PEAR is a supervised quality estimation metric that reframes reference-free machine translation evaluation as a graded pairwise comparison. |
| Approach: | They propose to use a supervised quality estimation metric family to reframe machine translation evaluation as a graded pairwise comparison. |
| Outcome: | The proposed metric outperforms strictly matched single-candidate QE baselines on the WMT24 meta-evaluation benchmark. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |