Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing tools for evaluation of translation models focus on high-level metrics like BLEU or COMET scores, which are time-consuming and prone to error. |
| Approach: | They propose a toolkit that provides a detailed analysis of translation models and a user-friendly interface. |
| Outcome: | The toolkit shows superior performance over COMET and SacreBLEU packages under enjoybility and understandbility criteria. |
Similar Papers
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics (2023.acl-short)
Copied to clipboard
| Challenge: | Neural metrics for machine translation evaluation are considered "black boxes" lexical overlap-based metrics are popular for evaluation of translation systems and algorithms . |
| Approach: | They develop and compare several neural explainability methods to understand translation errors . they aim to better understand the correspondence between token-level explanations and human annotated error spans . |
| Outcome: | The proposed methods leverage token-level information that can be directly attributed to translation errors. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020.emnlp-demos)
Copied to clipboard
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, Ann Yuan
| Challenge: | Existing tools for modeling and understanding models are limited . existing tools can assist practitioners in understanding and evaluating models . |
| Approach: | They present an open-source platform for visualization and understanding of NLP models. |
| Outcome: | The language interpretability tool (lit) is an open-source platform for visualization and understanding of NLP models. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
On Context Span Needed for Machine Translation Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of common patterns can be observed for context-aware MT evaluation, authors say . document-level evaluations have largely been performed at the sentence level . the definition of what constitutes a "document level" evaluation is still unclear . |
| Approach: | They propose to use a series of surveys to identify the necessary context span . they find common patterns that can be used to draw general guidelines . |
| Outcome: | The proposed evaluations of machine translation systems show that some issues and spans depend on domain and target language. |
An Evaluation Resource for Grounding Translation Errors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current fine-grained error analyses do not ground the errors to the reasons why the annotated text spans are erroneous. |
| Approach: | They use a bi-directional grounding scheme to ground erroneous text in two directions . if the error spans of both directions are consistent, the explanation is valid . |
| Outcome: | The proposed grounding process improves translation error detection significantly. |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
COMET for Low-Resource Machine Translation Evaluation: A Case Study of English-Maltese and Spanish-Basque (2024.lrec-main)
Copied to clipboard
| Challenge: | Trainable metrics for machine translation evaluation have been scoring the highest correlations with human judgements in the meta-evaluations. |
| Approach: | They run a crowd-based evaluation campaign to evaluate COMET-22 and fine-tune it to improve its performance. |
| Outcome: | The proposed system outperforms BLEU and other lexical overlap metrics in the meta-evaluations. |
COMET: A Neural Framework for MT Evaluation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Historically, metrics for evaluating the quality of machine translation (MT) have relied on basic, lexical-level features such as counting the number of matching n-grams between the MT hypothesis and the reference translation. |
| Approach: | They propose a neural framework for training multilingual machine translation evaluation models which exploits human judgements to obtain new state-of-the-art levels of correlation with MT quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems. |
ExplainaBoard: An Explainable Leaderboard for NLP (2021.acl-demo)
Copied to clipboard
Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Graham Neubig
| Challenge: | Using leaderboards, researchers can track the performance of various systems on various NLP tasks. |
| Approach: | They propose a new conceptualization and implementation of NLP evaluation using a leaderboard. |
| Outcome: | The ExplainaBoard is an evaluation tool for natural language processing (NLP) it covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. |