Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET (2022.aacl-main)
Copied to clipboard
| Challenge: | Neural metrics have a high correlation with human judgements but they are hard to eliminate due to their "black box" nature. |
| Approach: | They propose to use minimum bayes risk decoding to explore and quantify weaknesses in COMET models. |
| Outcome: | The proposed model is not sensitive enough to discrepancies in numbers and named entities, and is hard to remove by training on additional synthetic data. |
Similar Papers
Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) currently exhibits biases such as producing translations that are too short and overgenerating frequent words. |
| Approach: | They propose to use minimum bayes risk decoding instead of beam search to investigate the effects of beam decoding on unbiased samples. |
| Outcome: | The proposed method improves on a number of previously reported biases and failure cases of beam search on unbiased samples. |
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics (2022.tacl-1)
Copied to clipboard
| Challenge: | Neural machine translations are ranked below human translations in professional evaluations . |
| Approach: | They apply minimum bayes risk decoding to optimize different metrics of translation quality . they show that model estimates and translation quality only vaguely correlate . |
| Outcome: | The proposed method improves human translations with different models and metric. |
Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve beam search quality are inadequate in many ways . a new approximation to the beam search curse has been proposed . |
| Approach: | They propose an approximation to minimum Bayes risk decoding that would solve the beam search curse. |
| Outcome: | The proposed approximation has no equivalent to the beam search curse. |
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in machine translation (MT) have shown that minimum bayes risk decoding can be a powerful alternative to beam search. |
| Approach: | They propose to use epsilon-sampling to prune away all tokens with a smaller probability mass. |
| Outcome: | The proposed method outperforms beam search decoding and other methods in four languages. |
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics (2023.acl-short)
Copied to clipboard
| Challenge: | Neural metrics for machine translation evaluation are considered "black boxes" lexical overlap-based metrics are popular for evaluation of translation systems and algorithms . |
| Approach: | They develop and compare several neural explainability methods to understand translation errors . they aim to better understand the correspondence between token-level explanations and human annotated error spans . |
| Outcome: | The proposed methods leverage token-level information that can be directly attributed to translation errors. |
BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training (2023.acl-long)
Copied to clipboard
| Challenge: | Automated metrics are used for machine translation, but they are often considered to be black boxes with potential biases that are difficult to detect. |
| Approach: | They analyze automatic metrics from the perspective of their guidance for machine translation training. |
| Outcome: | The proposed measures improve the performance of machine translation models. |
Centroid-Based Efficient Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. |
| Approach: | They propose a centroid-based MBR decoding method that clusters the translations in the feature space and calculates the expected score using the centroids of each cluster. |
| Outcome: | The proposed method outperforms vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT’22 EnJa, EnDe, EnZh, and WMT'23 Enja translation tasks. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
COMET for Low-Resource Machine Translation Evaluation: A Case Study of English-Maltese and Spanish-Basque (2024.lrec-main)
Copied to clipboard
| Challenge: | Trainable metrics for machine translation evaluation have been scoring the highest correlations with human judgements in the meta-evaluations. |
| Approach: | They run a crowd-based evaluation campaign to evaluate COMET-22 and fine-tune it to improve its performance. |
| Outcome: | The proposed system outperforms BLEU and other lexical overlap metrics in the meta-evaluations. |
Can Automatic Metrics Assess High-Quality Translations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent human evaluation study found that translations produced by current MT systems achieve very high-quality scores when judged by humans on a direct assessment scale of 0 to 100. |
| Approach: | They stress-test the ability of current translation quality metrics to detect correct translations . they show that current metrics often over or underestimate translation quality . |
| Outcome: | The proposed method overestimates translation quality, the authors show . they show that current metrics often overestimate translation quality . |