MT-Telescope: An interactive platform for contrastive evaluation of MT systems (2021.acl-demo)
Copied to clipboard
| Challenge: | MT-Telescope is an open source, written in Python, and is built around a user friendly and dynamic web interface. |
| Approach: | They propose a platform to facilitate comparative analysis of the output quality of two Machine Translation (MT) systems. |
| Outcome: | The proposed platform supports fine-grained segment-level analysis and interactive visualisations that expose the fundamental differences in the performance of the compared systems. |
Similar Papers
MTLens: Machine Translation Output Debugging (2022.lrec-1)
Copied to clipboard
Shreyas Sharma, Kareem Darwish, Lucas Pavanelli, Thiago Castro Ferreira, Mohamed Al-Badrashiny, Kamer Ali Yuksel, Hassan Sawaf
| Challenge: | a demo demonstrates a system for quantitatively evaluating MT systems in isolation or multiple MT models collectively . performance of machine translation systems varies significantly with inputs of diverging features, such as genres, genres and surface properties. |
| Approach: | They propose a benchmarking interface that quantitatively evaluates MT systems in isolation or collectively . the interface can be extended to include additional filters such as lexical, morphological, and syntactic features. |
| Outcome: | The proposed system quantitatively evaluates MT systems on multiple domains and evaluation metrics. |
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (N19-4)
Copied to clipboard
| Challenge: | Unlike machine translation, natural language outputs are nuanced and there are no clear yes/no distinctions about whether they are correct or not. |
| Approach: | They describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. |
| Outcome: | The compare-mt tool is an open-source pure-python package that has already proven useful to generate analyses that have been used in our papers. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers (2021.acl-long)
Copied to clipboard
| Challenge: | a meta-evaluation of machine translation (MT) has been conducted in 769 research papers . a recent study shows that evaluation practices have changed over the past decade . |
| Approach: | They propose a meta-evaluation method for machine translation that uses BLEU scores to evaluate MT performance. |
| Outcome: | The proposed meta-evaluation of machine translation shows that evaluation practices have changed over the past decade . the authors suggest that the evaluation process should be streamlined and standardized to ensure the validity of the evaluation method . |
Tilde MT Platform for Developing Client Specific MT Solutions (L18-1)
Copied to clipboard
| Challenge: | a growing demand for translations and multilingual content is surpassing the supply of professional translation services. |
| Approach: | They present a custom machine translation platform called Tilde MT that provides linguistic data storage, data cleaning and normalisation, statistical and neural machine translation system training and hosting functionality. |
| Outcome: | The proposed platform provides linguistic data storage, data cleaning and normalisation, statistical and neural machine translation system training and hosting functionality, and wide integration capabilities. |
The OPUS-MT Dashboard – A Toolkit for a Systematic Evaluation of Open Machine Translation Models (2023.acl-demo)
Copied to clipboard
| Challenge: | OPUS-MT dashboard provides a comprehensive overview of open translation models . the landscape of machine translation (MT) is increasingly blurry due to the growing volume of shared tasks and models published within the community. |
| Approach: | OPUS-MT dashboard provides a comprehensive overview of open translation models . dashboard includes summaries of benchmarks for over 2,300 models covering 4,560 languages . authors focus on centralization, reproducibility and coverage of MT evaluation combined with scalability . |
| Outcome: | OPUS-MT dashboard provides a comprehensive overview of open translation models . the evaluation tool includes summaries of benchmarks for over 2,300 models spanning 4,560 languages and 294 languages . |
Informative Manual Evaluation of Machine Translation Output (2020.coling-main)
Copied to clipboard
| Challenge: | a new method for manual evaluation of machine translation output is proposed . evaluators mark problematic parts of the translated text, not just overall scores . |
| Approach: | They propose a method for manual evaluation of machine translation output based on marking actual issues in the translated text. |
| Outcome: | The proposed method can be applied on any genre/domain and language pair . it can be guided by various types of quality criteria and can be used for other types of generated text. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
Extrinsic Evaluation of Machine Translation Metrics (2023.acl-long)
Copied to clipboard
| Challenge: | MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
| Approach: | They evaluate the segment-level performance of the most widely used MT metrics by correlating them with how useful they are for downstream tasks. |
| Outcome: | The MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |