Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | a genetic algorithm (GA) based method improves MT quality and identifies weaknesses in evaluation metrics. |
| Approach: | They propose a genetic algorithm-based method for modifying n-best lists produced by a machine translation system using a fitness function. |
| Outcome: | The proposed method improves translation quality and identifies weaknesses in evaluation metrics. |
Similar Papers
GAATME: A Genetic Algorithm for Adversarial Translation Metrics Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | a genetic algorithm generates adversarial translations for arbitrary metrics . produced translations score well in an arbitrary MT evaluation metric, despite serious, deliberately introduced errors. |
| Approach: | They propose a method for decoding translation candidates from a machine translation model via a genetic algorithm to generate adversarial translations to test and challenge MT evaluation metrics. |
| Outcome: | The proposed method scores very well in an arbitrary MT evaluation metric despite serious, deliberately introduced errors. |
Evaluating Automatic Metrics with Incremental Machine Translation Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that neural metrics are more reliable than non-neural metrics. |
| Approach: | They propose to use commercial machine translations to evaluate machine translation metrics based on their preference for more recent outputs. |
| Outcome: | The proposed dataset confirms several previous findings, including the advantage of neural metrics over non-neural ones, and also explores the debated issue of how MT quality affects metric reliability. |
Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | BERTScore, BLEURT, and COMET are automatic evaluation metrics that are often underperformed on adversarially-synthesized texts. |
| Approach: | They examine MT evaluation metric performance on adversarially-synthesized texts . they validate that automatic metrics tend to overpenalize adversarial-degraded translations . |
| Outcome: | The results show that automatic metrics tend to overpenalize adversarially-degraded translations. |
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate multiple systems are expensive and require human evaluators. |
| Approach: | They propose a novel online learning approach that dynamically converges to the top-3 ranked systems for the language pairs considered by taking advantage of human feedback. |
| Outcome: | The proposed approach converges to the top-3 ranked systems for the language pairs considered despite the lack of human feedback for many translations. |
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
Multi-Hypothesis Machine Translation Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. |
| Approach: | They propose to use MT models to generate multiple diverse translations and use them as surrogates to reference translations to obtain a quantification of translation variability. |
| Outcome: | The proposed approach improves correlation with human judgements of quality by 15%. |
Automated Paraphrase Lattice Creation for HyTER Machine Translation Evaluation (N18-2)
Copied to clipboard
| Challenge: | Existing machine translation evaluation metrics use synonyms and paraphrases to reward meaning-equivalent but lexically divergent translations. |
| Approach: | They propose a machine translation evaluation metric which exploits reference translations enriched with meaning equivalent expressions. |
| Outcome: | The proposed metric achieves medium performance on large and noisier datasets . it is compared with the existing HyTER evaluation metric . |
Machine Translation into Low-resource Language Varieties (2021.acl-short)
Copied to clipboard
| Challenge: | Current machine translation systems generate a "standard" target language, but many languages have multiple varieties that are different from the standard language. |
| Approach: | They propose a framework to rapidly adapt machine translation systems to generate different target varieties . they propose to use no parallel data to generate languages close to, but different from, the standard target language . |
| Outcome: | The proposed model improves on a system that generates Ukrainian and Belarusian in two languages with no parallel data. |
EvolveMT: an Ensemble MT Engine Improving Itself with Usage Only (2023.acl-industry)
Copied to clipboard
| Challenge: | EvolveMT is a method for the efficient combination of multiple machine translation engines. |
| Approach: | They propose a method that selects the output from one engine for each segment and uses online learning techniques to predict the most appropriate system for each translation request. |
| Outcome: | The proposed method achieves similar translation accuracy at a lower cost than selecting the best translation of each segment from all translations using an MT quality estimator. |
Generating Diverse Translation with Perturbed kNN-MT (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods to generate multiple translation candidates do not address the overcorrection problem, which discourages the model from generating synonymous expressions and leans toward gold standards, reducing the diversity in the candidates. |
| Approach: | They propose to introduce perturbed k-nearest neighbor machine translation (kNN-MT) to generate more diverse translations. |
| Outcome: | The proposed methods significantly improve candidate diversity and control diversity by tuning the perturbation’s magnitude. |