Evaluation of Machine Translation Performance Across Multiple Genres and Languages (L18-1)
Copied to clipboard
| Challenge: | In this paper, we evaluate the impact of genre differences on machine translation (MT) for a diverse set of language pairs . BLEU score differences between genres can be large for all genres and all language pairs. |
| Approach: | They use multi-genre benchmarks to evaluate the impact of genre differences on machine translation (MT) they train and use genre classifiers to route test documents to the most appropriate genre systems . |
| Outcome: | The proposed system can improve translation quality for all genres and language pairs . |
Similar Papers
A Benchmark for Translations Across Styles and Language Variants (2025.findings-emnlp)
Copied to clipboard
| Challenge: | lack of comprehensive evaluation benchmarks has hindered progress in this field . lack of evaluation benchmarking has hinder MT's ability to generate accurate outputs . |
| Approach: | They evaluate translations across semantic preservation, cultural and regional specificity, expression style, and fluency at both the word and sentence levels. |
| Outcome: | The proposed evaluation framework is validated on translations of state-of-the-art large language models . |
Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark (2025.emnlp-main)
Copied to clipboard
Chihiro Taguchi, Seng Mai, Keita Kurabe, Yusuke Sakai, Georgina Agyei, Soudabeh Eslami, David Chiang
| Challenge: | Multilingual machine translation (MT) benchmarks are widely used to evaluate the capabilities of modern MT systems. |
| Approach: | They propose to use a multilingual machine translation benchmark to assess the capabilities of modern machine translation systems. |
| Outcome: | The FLORES+ benchmark claims to maintain a translation quality score of over 90% . however, the data in four languages falls short of the 90% quality standard . |
MTLens: Machine Translation Output Debugging (2022.lrec-1)
Copied to clipboard
Shreyas Sharma, Kareem Darwish, Lucas Pavanelli, Thiago Castro Ferreira, Mohamed Al-Badrashiny, Kamer Ali Yuksel, Hassan Sawaf
| Challenge: | a demo demonstrates a system for quantitatively evaluating MT systems in isolation or multiple MT models collectively . performance of machine translation systems varies significantly with inputs of diverging features, such as genres, genres and surface properties. |
| Approach: | They propose a benchmarking interface that quantitatively evaluates MT systems in isolation or collectively . the interface can be extended to include additional filters such as lexical, morphological, and syntactic features. |
| Outcome: | The proposed system quantitatively evaluates MT systems on multiple domains and evaluation metrics. |
Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on MT evaluation characterize quality of output with a single number . a recent advancement in MT technologies has enabled higher-quality, more nuanced translations . |
| Approach: | They propose a 1200-sentence MQM evaluation benchmark for English-Korean and a reference-free QE setup to evaluate the quality of the translations. |
| Outcome: | The proposed model outperforms the existing model in style and accuracy. |
Informative Manual Evaluation of Machine Translation Output (2020.coling-main)
Copied to clipboard
| Challenge: | a new method for manual evaluation of machine translation output is proposed . evaluators mark problematic parts of the translated text, not just overall scores . |
| Approach: | They propose a method for manual evaluation of machine translation output based on marking actual issues in the translated text. |
| Outcome: | The proposed method can be applied on any genre/domain and language pair . it can be guided by various types of quality criteria and can be used for other types of generated text. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has focused on literary machine translation (MT) but evaluation of literary MT remains an open problem. |
| Approach: | They propose a paragraph-level parallel corpus containing verified human translations and 13k evaluated sentences across four language pairs. |
| Outcome: | The proposed corpus compares human evaluations with students and professionals . it shows that the adequacy of human evaluation is controlled by two factors . |
Evaluating Domain Adaptation for Machine Translation Across Scenarios (L18-1)
Copied to clipboard
| Challenge: | Statistical machine translation (SMT) has been the dominant approach for the last 20 years, with neural machine translation becoming the new main paradigm in academic research and the industry. |
| Approach: | They propose to compare domain-adapted statistical and neural machine translation systems on three different domains and language pairs with varying degrees of domain specificity and available training data. |
| Outcome: | The proposed system is the best choice for translation, with marked impacts for domains with higher specificity. |
Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)
Copied to clipboard
| Challenge: | Several recent papers claim to have achieved human parity at sentence-level machine translation. |
| Approach: | They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures. |
| Outcome: | The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations . |