COMET for Low-Resource Machine Translation Evaluation: A Case Study of English-Maltese and Spanish-Basque (2024.lrec-main)
Copied to clipboard
| Challenge: | Trainable metrics for machine translation evaluation have been scoring the highest correlations with human judgements in the meta-evaluations. |
| Approach: | They run a crowd-based evaluation campaign to evaluate COMET-22 and fine-tune it to improve its performance. |
| Outcome: | The proposed system outperforms BLEU and other lexical overlap metrics in the meta-evaluations. |
Similar Papers
COMET: A Neural Framework for MT Evaluation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Historically, metrics for evaluating the quality of machine translation (MT) have relied on basic, lexical-level features such as counting the number of matching n-grams between the MT hypothesis and the reference translation. |
| Approach: | They propose a neural framework for training multilingual machine translation evaluation models which exploits human judgements to obtain new state-of-the-art levels of correlation with MT quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems. |
COMET-QE and Active Learning for Low-Resource Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using COMET-QE, we select sentences for low-resource neural machine translation. |
| Approach: | They propose a reference-free evaluation metric to select sentences for low-resource neural machine translation using Swahili, Kinyarwanda and Spanish. |
| Outcome: | The proposed method outperforms two variants of Round Trip Translation Likelihood and random sentence selection by up to 5 BLEU points on a 30k baseline. |
SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? (2025.emnlp-main)
Copied to clipboard
Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, David Ifeoluwa Adelani
| Challenge: | Existing metrics for machine translation quality for under-resourced African languages suffer from limited language coverage and poor performance in low-resource settings. |
| Approach: | They propose a large-scale human-annotated machine translation evaluation dataset . they use a reference-based and reference-free evaluation model to compare MT quality . |
| Outcome: | The proposed models outperform AfriCOMET and the strongest LLM on low-resource languages. |
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics (2023.acl-short)
Copied to clipboard
| Challenge: | Neural metrics for machine translation evaluation are considered "black boxes" lexical overlap-based metrics are popular for evaluation of translation systems and algorithms . |
| Approach: | They develop and compare several neural explainability methods to understand translation errors . they aim to better understand the correspondence between token-level explanations and human annotated error spans . |
| Outcome: | The proposed methods leverage token-level information that can be directly attributed to translation errors. |
Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET (2022.aacl-main)
Copied to clipboard
| Challenge: | Neural metrics have a high correlation with human judgements but they are hard to eliminate due to their "black box" nature. |
| Approach: | They propose to use minimum bayes risk decoding to explore and quantify weaknesses in COMET models. |
| Outcome: | The proposed model is not sensitive enough to discrepancies in numbers and named entities, and is hard to remove by training on additional synthetic data. |
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages (2023.acl-long)
Copied to clipboard
Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre
| Challenge: | Recent studies on machine translation systems focus on high-resource languages, but focus has shifted to low-resourced languages. |
| Approach: | They evaluate 16 metrics from a multidimensional quality metric dataset . they show pre-trained metrics have higher correlations with annotator scores . |
| Outcome: | The proposed evaluations show that pre-trained metrics outperform COMET on Indian languages. |
AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages (2024.naacl-long)
Copied to clipboard
Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Mohamed, Hassan Ayinde, Oluwabusayo Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Toadoum Sari Sakayo, Lyse Naomi Wamba, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Iro, Saheed Abdullahi, Stephen Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Ogbu, Sam Ochieng’, Verrah Otiende, Chinedu Mbonu, Yao Lu, Pontus Stenetorp
| Challenge: | Recent advances in machine translation (MT) have focused on scaling multilingual machine translation models and evaluation data to hundreds of languages, including multiple under-resourced languages. |
| Approach: | They propose to use n-gram matching metrics to measure progress in multilingual machine translation to 13 typologically diverse African languages to create high-quality human evaluation data with simplified MQM guidelines. |
| Outcome: | The proposed metrics have a higher correlation with human judgments than n-gram matching metrics such as BLEU and METEOR. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Exploring Context-Aware Evaluation Metrics for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on machine translation evaluation focused on quality of individual sentences, while neglecting the importance of contextual information. |
| Approach: | They propose a context-aware machine translation evaluation metric called Cont-COMET . they use the COMET framework to consider the preceding and subsequent contexts of the sentence . |
| Outcome: | The proposed metric improves system-level and segment-level evaluations on the official WMT framework. |
Simul-COMET: A Quality Metric for Simultaneous Interpretation in Distant Language Pair Considering Word Order Difference (2026.findings-acl)
Copied to clipboard
| Challenge: | Simultaneous interpretation (SI) uses segmenting of source speech into chunks and translating them in order. |
| Approach: | They propose a variation of COMET that measures monotonicity for simultaneous interpretation . they train Simul-COMET on offline translation data and show stronger alignment with evaluation scores . |
| Outcome: | The proposed model shows stronger alignment with evaluation scores provided by interpreters than COMET. |