Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite (D19-1)
Copied to clipboard
| Challenge: | Currently, machine translation is performed at the level of individual sentences, in isolation from the rest of the document. |
| Approach: | They propose a dataset that can be used as a test suite for pronoun translation . they propose an evaluation measure to differentiate good and bad pronounce translations . |
| Outcome: | The proposed dataset can be used as a test suite for pronoun translation in English . it covers multiple source languages and different pronouner errors drawn from real system translations . |
Similar Papers
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point (D18-1)
Copied to clipboard
| Challenge: | a range of issues limit the performance of the automated metrics. |
| Approach: | They propose to use semi-automatic metrics and test suites instead of fully automatic metrics for pronoun translation. |
| Outcome: | The proposed metrics improve translation accuracy by comparing them against a manually annotated dataset . the proposed metrics are semi-automatic and test suites in place of fully automatic metrics. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
Evaluation Dataset for Zero Pronoun in Japanese to English Translation (2020.lrec-1)
Copied to clipboard
| Challenge: | In natural language, we often omit some words that are easily understandable from the context. |
| Approach: | They propose to use a dataset to evaluate whether translation models can resolve zero pronoun problems in Japanese to English translations. |
| Outcome: | The proposed model can resolve the zero pronoun problem in Japanese to English translations. |
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are limited and can be easily portable to new languages. |
| Approach: | They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. |
| Outcome: | The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models. |
On Context Span Needed for Machine Translation Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of common patterns can be observed for context-aware MT evaluation, authors say . document-level evaluations have largely been performed at the sentence level . the definition of what constitutes a "document level" evaluation is still unclear . |
| Approach: | They propose to use a series of surveys to identify the necessary context span . they find common patterns that can be used to draw general guidelines . |
| Outcome: | The proposed evaluations of machine translation systems show that some issues and spans depend on domain and target language. |
How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts. (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation systems are prone to gender biases in their learned representations. |
| Approach: | They propose to use contextual sentences to correct gender bias in Neural Machine Translation models. |
| Outcome: | The proposed method can be used to build better, bias-free translation systems. |
Context-Aware Neural Machine Translation Learns Anaphora Resolution (P18-1)
Copied to clipboard
| Challenge: | Standard machine translation systems process sentences in isolation and ignore extra-sentential information. |
| Approach: | They propose a context-aware neural machine translation model that controls flow of information from extended context to the translation model. |
| Outcome: | The proposed model improves on an English-Russian subtitles dataset over its context-agnostic version (+0.7) and over simple concatenation of context and source sentences (+0.6). |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
Evaluating Automatic Metrics with Incremental Machine Translation Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that neural metrics are more reliable than non-neural metrics. |
| Approach: | They propose to use commercial machine translations to evaluate machine translation metrics based on their preference for more recent outputs. |
| Outcome: | The proposed dataset confirms several previous findings, including the advantage of neural metrics over non-neural ones, and also explores the debated issue of how MT quality affects metric reliability. |
Do Context-Aware Translation Models Pay the Right Attention? (2021.acl-long)
Copied to clipboard
| Challenge: | Context-aware machine translation models fail to leverage contextual information to resolve ambiguous words and pronouns. |
| Approach: | They propose a new dataset that includes supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. |
| Outcome: | The proposed model can automatically disambiguate pronouns and polysemous words when they are not in the same context. |