Exploring Context-Aware Evaluation Metrics for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on machine translation evaluation focused on quality of individual sentences, while neglecting the importance of contextual information. |
| Approach: | They propose a context-aware machine translation evaluation metric called Cont-COMET . they use the COMET framework to consider the preceding and subsequent contexts of the sentence . |
| Outcome: | The proposed metric improves system-level and segment-level evaluations on the official WMT framework. |
Similar Papers
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are limited and can be easily portable to new languages. |
| Approach: | They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. |
| Outcome: | The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models. |
COMET: A Neural Framework for MT Evaluation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Historically, metrics for evaluating the quality of machine translation (MT) have relied on basic, lexical-level features such as counting the number of matching n-grams between the MT hypothesis and the reference translation. |
| Approach: | They propose a neural framework for training multilingual machine translation evaluation models which exploits human judgements to obtain new state-of-the-art levels of correlation with MT quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems. |
On Context Span Needed for Machine Translation Evaluation (2020.lrec-1)
Copied to clipboard
| Challenge: | a number of common patterns can be observed for context-aware MT evaluation, authors say . document-level evaluations have largely been performed at the sentence level . the definition of what constitutes a "document level" evaluation is still unclear . |
| Approach: | They propose to use a series of surveys to identify the necessary context span . they find common patterns that can be used to draw general guidelines . |
| Outcome: | The proposed evaluations of machine translation systems show that some issues and spans depend on domain and target language. |
Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)
Copied to clipboard
Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi
| Challenge: | a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics . |
| Approach: | They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context. |
| Outcome: | The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements. |
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021.tacl-1)
Copied to clipboard
| Challenge: | a large study of machine translation systems shows poor evaluation procedures can lead to erroneous conclusions. |
| Approach: | They propose an evaluation methodology grounded in explicit error analysis based on the Multidimensional Quality Metrics framework. |
| Outcome: | The proposed evaluation methodology outperforms crowd workers in two languages . it shows that human-based metrics outperformed crowd workers . |
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics (2023.acl-short)
Copied to clipboard
| Challenge: | Neural metrics for machine translation evaluation are considered "black boxes" lexical overlap-based metrics are popular for evaluation of translation systems and algorithms . |
| Approach: | They develop and compare several neural explainability methods to understand translation errors . they aim to better understand the correspondence between token-level explanations and human annotated error spans . |
| Outcome: | The proposed methods leverage token-level information that can be directly attributed to translation errors. |
Measuring and Increasing Context Usage in Context-Aware Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in neural machine translation has demonstrated the necessity and feasibility of using inter-sentential context, but it is often not clear how much they actually utilize it at translation time. |
| Approach: | They propose a conditional cross-mutual information metric to quantify usage of context by model architectures that can use it at translation time. |
| Outcome: | The proposed method increases context usage and improves translation quality according to BLEU and COMET metrics. |
Revisiting Context Choices for Context-aware Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work has cast doubt on whether context-aware machine translation models learn useful signals from context or are improvements in automatic evaluation metrics just a side-effect. |
| Approach: | They propose to use separate encoders for source sentence and context as multiple sources for one target sentence to train context-aware machine translation models. |
| Outcome: | The proposed model improves translation quality even with empty lines as context, but the correct context improves it and random out-of-domain context degrades it. |
COMET for Low-Resource Machine Translation Evaluation: A Case Study of English-Maltese and Spanish-Basque (2024.lrec-main)
Copied to clipboard
| Challenge: | Trainable metrics for machine translation evaluation have been scoring the highest correlations with human judgements in the meta-evaluations. |
| Approach: | They run a crowd-based evaluation campaign to evaluate COMET-22 and fine-tune it to improve its performance. |
| Outcome: | The proposed system outperforms BLEU and other lexical overlap metrics in the meta-evaluations. |
Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations (N18-4)
Copied to clipboard
| Challenge: | Sentence representations can capture information that cannot be captured by local features based on character or word Ngrams. |
| Approach: | They propose a supervised regression model using universal sentence representations capable of capturing information that cannot be captured by local features based on character or word Ngrams. |
| Outcome: | The proposed model achieves state-of-the-art performance with only sentence representation features . |