Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation (D18-1)
Copied to clipboard
| Challenge: | Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese–English news translation task. |
| Approach: | They empirically test neural machine translation on a Chinese–English news translation task . they show human raters prefer human over machine translation when evaluating documents . |
| Outcome: | The proposed method shows that human translators prefer document-level evaluation over machine translation . the results highlight the need to shift towards document- level evaluation as machine translation improves . |
Similar Papers
On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | In this paper, we reassess claims of human parity and super human performance in machine translation. |
| Approach: | They reassess claims of human parity and super human performance in machine translation . they argue that human translation involves much more than what is embedded in automatic systems . |
| Outcome: | The proposed results show that human translation involves much more than what is embedded in automatic systems. |
Assessing Human-Parity in Machine Translation on the Segment Level (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent machine translation shared tasks have shown top-performing systems to tie or outperform human translation. |
| Approach: | They examine the outputs of top-performing systems in a recent machine translation shared task . they find that some systems outperform human translation on average . |
| Outcome: | a new method identifies segments for which human and machine perform poorly . the results show that top-performing systems outperform human translation on average . |
Rethinking Document-level Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation models are weak enough for document-level translation . current models only translate sentences individually, resulting in poor document coherence . |
| Approach: | They propose to use the original Transformer model to test document-level neural machine translation . they find that the original transformer models can achieve strong results for document translation if trained properly . |
| Outcome: | The proposed model outperforms sentence-level models on nine datasets and two sentence- level datasets across six languages. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |
Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)
Copied to clipboard
| Challenge: | Several recent papers claim to have achieved human parity at sentence-level machine translation. |
| Approach: | They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures. |
| Outcome: | The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations . |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
Upping the Ante: Towards a Better Benchmark for Chinese-to-English Machine Translation (L18-1)
Copied to clipboard
| Challenge: | Currently, there is no widely accepted standard for evaluation of machine translation (MT) for Chinese-to-English translation, there are no standard for standardized training sets, development sets, and test sets. |
| Approach: | They propose to use Chinese-to-English machine translation as a benchmark . they build a highly competitive state-of-the-art MT system that outperforms reported results . |
| Outcome: | The proposed system outperforms reported results on NIST OpenMT test sets in almost all papers published in major conferences and journals in computational linguistics and artificial intelligence in the past 11 years. |
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate multiple systems are expensive and require human evaluators. |
| Approach: | They propose a novel online learning approach that dynamically converges to the top-3 ranked systems for the language pairs considered by taking advantage of human feedback. |
| Outcome: | The proposed approach converges to the top-3 ranked systems for the language pairs considered despite the lack of human feedback for many translations. |
How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has focused on literary machine translation (MT) but evaluation of literary MT remains an open problem. |
| Approach: | They propose a paragraph-level parallel corpus containing verified human translations and 13k evaluated sentences across four language pairs. |
| Outcome: | The proposed corpus compares human evaluations with students and professionals . it shows that the adequacy of human evaluation is controlled by two factors . |
On Context Span Needed for Machine Translation Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of common patterns can be observed for context-aware MT evaluation, authors say . document-level evaluations have largely been performed at the sentence level . the definition of what constitutes a "document level" evaluation is still unclear . |
| Approach: | They propose to use a series of surveys to identify the necessary context span . they find common patterns that can be used to draw general guidelines . |
| Outcome: | The proposed evaluations of machine translation systems show that some issues and spans depend on domain and target language. |