compare-mt: A Tool for Holistic Comparison of Language Generation Systems (N19-4)
Copied to clipboard
| Challenge: | Unlike machine translation, natural language outputs are nuanced and there are no clear yes/no distinctions about whether they are correct or not. |
| Approach: | They describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. |
| Outcome: | The compare-mt tool is an open-source pure-python package that has already proven useful to generate analyses that have been used in our papers. |
Similar Papers
MT-Telescope: An interactive platform for contrastive evaluation of MT systems (2021.acl-demo)
Copied to clipboard
| Challenge: | MT-Telescope is an open source, written in Python, and is built around a user friendly and dynamic web interface. |
| Approach: | They propose a platform to facilitate comparative analysis of the output quality of two Machine Translation (MT) systems. |
| Outcome: | The proposed platform supports fine-grained segment-level analysis and interactive visualisations that expose the fundamental differences in the performance of the compared systems. |
Machine Translation into Low-resource Language Varieties (2021.acl-short)
Copied to clipboard
| Challenge: | Current machine translation systems generate a "standard" target language, but many languages have multiple varieties that are different from the standard language. |
| Approach: | They propose a framework to rapidly adapt machine translation systems to generate different target varieties . they propose to use no parallel data to generate languages close to, but different from, the standard target language . |
| Outcome: | The proposed model improves on a system that generates Ukrainian and Belarusian in two languages with no parallel data. |
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020.coling-main)
Copied to clipboard
| Challenge: | a few popular metrics are still used to evaluate language generation systems despite their known limitations. |
| Approach: | They propose to use automatic metrics to evaluate language generation systems . they show that they prefer system outputs to human-authored texts . |
| Outcome: | The proposed metrics are insensitive to correct translations of rare words and can yield high scores when given a single sentence as system output for the entire test set. |
Informative Manual Evaluation of Machine Translation Output (2020.coling-main)
Copied to clipboard
| Challenge: | a new method for manual evaluation of machine translation output is proposed . evaluators mark problematic parts of the translated text, not just overall scores . |
| Approach: | They propose a method for manual evaluation of machine translation output based on marking actual issues in the translated text. |
| Outcome: | The proposed method can be applied on any genre/domain and language pair . it can be guided by various types of quality criteria and can be used for other types of generated text. |
A Benchmark for Translations Across Styles and Language Variants (2025.findings-emnlp)
Copied to clipboard
| Challenge: | lack of comprehensive evaluation benchmarks has hindered progress in this field . lack of evaluation benchmarking has hinder MT's ability to generate accurate outputs . |
| Approach: | They evaluate translations across semantic preservation, cultural and regional specificity, expression style, and fluency at both the word and sentence levels. |
| Outcome: | The proposed evaluation framework is validated on translations of state-of-the-art large language models . |
LMdiff: A Visual Diff Tool to Compare Language Models (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LMdiff visually compares probability distributions of two different language models . notably absent from the range of available tools are those that aim to compare distributions produced by different models. |
| Approach: | They propose a tool that visually compares probability distributions of two different language models that differ through finetuning, distillation, or simply training with different parameter sizes. |
| Outcome: | The proposed tool allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing interesting text instances from large corpora. |
Tilde MT Platform for Developing Client Specific MT Solutions (L18-1)
Copied to clipboard
| Challenge: | a growing demand for translations and multilingual content is surpassing the supply of professional translation services. |
| Approach: | They present a custom machine translation platform called Tilde MT that provides linguistic data storage, data cleaning and normalisation, statistical and neural machine translation system training and hosting functionality. |
| Outcome: | The proposed platform provides linguistic data storage, data cleaning and normalisation, statistical and neural machine translation system training and hosting functionality, and wide integration capabilities. |
MTLens: Machine Translation Output Debugging (2022.lrec-1)
Copied to clipboard
Shreyas Sharma, Kareem Darwish, Lucas Pavanelli, Thiago Castro Ferreira, Mohamed Al-Badrashiny, Kamer Ali Yuksel, Hassan Sawaf
| Challenge: | a demo demonstrates a system for quantitatively evaluating MT systems in isolation or multiple MT models collectively . performance of machine translation systems varies significantly with inputs of diverging features, such as genres, genres and surface properties. |
| Approach: | They propose a benchmarking interface that quantitatively evaluates MT systems in isolation or collectively . the interface can be extended to include additional filters such as lexical, morphological, and syntactic features. |
| Outcome: | The proposed system quantitatively evaluates MT systems on multiple domains and evaluation metrics. |
CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Neural machine translation systems exhibit limited robustness in handling source-side linguistic variations. |
| Approach: | They propose a dialectal benchmark to quantify the robustness of MT systems to handle source-side linguistic variations. |
| Outcome: | The proposed benchmark demonstrates that large MT models face challenges translating dialectal variants. |
Which is Better for Deep Learning: Python or MATLAB? Answering Comparative Questions in Natural Language (2021.eacl-demos)
Copied to clipboard
Viktoriia Chekalina, Alexander Bondarenko, Chris Biemann, Meriem Beloucif, Varvara Logacheva, Alexander Panchenko
| Challenge: | Comparative QA is a challenging task since it requires collecting evidence from many different sources. |
| Approach: | They propose a natural language interface for comparative QA that can be used in personal assistants, chatbots, and similar NLP devices. |
| Outcome: | The proposed system can be used in personal assistants, chatbots, and similar NLP devices. |