MQM-Chat: Multidimensional Quality Metrics for Chat Translation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for chat translation face challenges due to high levels of ambiguity and stylized contents. |
| Approach: | They propose a multidimensional quality metric for chat translation that includes seven error types . they use human annotations to analyze chat data generated by five translation models . |
| Outcome: | The proposed evaluation metric can qualify errors while highlighting chat-specific issues explicitly. |
Similar Papers
LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing MT evaluation frameworks fail to capture dialect- and culture-specific errors in diglossic languages. |
| Approach: | They propose a hierarchical error taxonomy for diagnosing MT errors through six linguistic levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics. |
| Outcome: | The proposed framework produces 6,113 labeled error spans across 3,495 unique erroneous sentences . it is language-agnostic and can be easily applied to or adapted for other languages. |
Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on MT evaluation characterize quality of output with a single number . a recent advancement in MT technologies has enabled higher-quality, more nuanced translations . |
| Approach: | They propose a 1200-sentence MQM evaluation benchmark for English-Korean and a reference-free QE setup to evaluate the quality of the translations. |
| Outcome: | The proposed model outperforms the existing model in style and accuracy. |
Can Automatic Metrics Assess High-Quality Translations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent human evaluation study found that translations produced by current MT systems achieve very high-quality scores when judged by humans on a direct assessment scale of 0 to 100. |
| Approach: | They stress-test the ability of current translation quality metrics to detect correct translations . they show that current metrics often over or underestimate translation quality . |
| Outcome: | The proposed method overestimates translation quality, the authors show . they show that current metrics often overestimate translation quality . |
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021.tacl-1)
Copied to clipboard
| Challenge: | a large study of machine translation systems shows poor evaluation procedures can lead to erroneous conclusions. |
| Approach: | They propose an evaluation methodology grounded in explicit error analysis based on the Multidimensional Quality Metrics framework. |
| Outcome: | The proposed evaluation methodology outperforms crowd workers in two languages . it shows that human-based metrics outperformed crowd workers . |
Enhancing Human Evaluation in Machine Translation with Comparative Judgement (2025.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. |
| Approach: | They evaluate three annotation setups to integrate comparative judgment into human annotation for machine translation. |
| Outcome: | The proposed approach improves inter-annotator agreement and stability of the annotations. |
XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics (2026.findings-acl)
Copied to clipboard
| Challenge: | averaging metric scores across languages is suspicious since translations of equal quality receive different scores across language. |
| Approach: | They propose a semi-automatically built dataset to benchmark translation metrics using MQM-defined errors and a normalization strategy to mitigate cross-lingual scoring bias. |
| Outcome: | The proposed model shows that translation metrics suffer from cross-lingual scoring bias . the proposed model is based on a semi-automatically built dataset covering nine translation directions . |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern WQE techniques rely on expensive inference with large language models or ad-hoc training with large amounts of human-labeled data. |
| Approach: | They propose to use word-level quality estimation to identify translation errors from the inner workings of translation models to quantify the impact of human label variation on metric performance. |
| Outcome: | The proposed methods identify translation errors from the inner workings of translation models using human labels. |
Machine translation Evaluation Eng-Thai MQM Ranking dataset (2026.eacl-short)
Copied to clipboard
Phichet Phuangrot, Natdanai Trintawat, Kanawat Vilasri, Yanapat Patcharawiwatpong, Pachara Boonsarngsuk, Nat Pavasant, Ekapol Chuangsuwanich
| Challenge: | MEET-MR provides a comprehensive benchmark for evaluating English–Thai machine translation systems. |
| Approach: | They propose a benchmark for evaluating English–Thai machine translation systems . they use the Multidimensional Quality Metrics framework to provide fine-grained human judgements of translation quality. |
| Outcome: | The dataset covers nine domains providing linguistic and contextual diversity. |
Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains (2024.acl-short)
Copied to clipboard
| Challenge: | a new dataset examines whether fine-tuned metrics are robust to domain shifts between training and inference. |
| Approach: | They use an annotated multidimensional quality metrics dataset to examine whether they are robust to domain shifts between training and inference. |
| Outcome: | The proposed metrics exhibit a substantial performance drop in the unseen domain scenario compared to metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments. |