BiVert: Bidirectional Vocabulary Evaluation Using Relations for Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) evaluation is crucial to determine the quality of translations. |
| Approach: | They propose a bidirectional semantic-based evaluation method to assess the sense distance of the translation from the source text. |
| Outcome: | The proposed method uses the multilingual encyclopedic dictionary BabelNet . it shows a strong correlation between the evaluation scores and human assessments . |
Similar Papers
An Evaluation Benchmark for Testing the Word Sense Disambiguation Capabilities of Machine Translation Systems (2020.lrec-1)
Copied to clipboard
| Challenge: | Lexical ambiguity is one of the many challenging linguistic phenomena involved in translation, i.e., translating an ambiguous word with its correct sense. |
| Approach: | They propose to use training data to measure the sense distributions of a machine translation system to measure lexical ambiguity. |
| Outcome: | The proposed benchmark builds upon the multilingual sense inventory of BabelNet, the multilinguistic neural parsing pipeline TurkuNLP, and the OPUS collection of translated texts from the web. |
SentSim: Crosslingual Semantic Evaluation of Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Machine translation (MT) is currently evaluated in one of two ways: monolingually or trained crosslingually by building a supervised model to predict quality scores from human-labeled data. |
| Approach: | They propose an unsupervised model that directly compares the source and machine translated sentence using strong pretrained multilingual word and sentence representations. |
| Outcome: | The proposed model outperforms glass-box approaches to quality estimation that rely on a supervised model. |
Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on MT evaluation characterize quality of output with a single number . a recent advancement in MT technologies has enabled higher-quality, more nuanced translations . |
| Approach: | They propose a 1200-sentence MQM evaluation benchmark for English-Korean and a reference-free QE setup to evaluate the quality of the translations. |
| Outcome: | The proposed model outperforms the existing model in style and accuracy. |
CLIReval: Evaluating Machine Translation as a Cross-Lingual Information Retrieval Task (2020.acl-demos)
Copied to clipboard
| Challenge: | evaluating machine translation (MT) with cross-lingual information retrieval is relatively time-consuming and subjective. |
| Approach: | They propose a toolkit that evaluates machine translation with a proxy task of cross-lingual information retrieval. |
| Outcome: | The proposed toolkit is based on the "metrics shared task" of WMT2019. |
An Effective Approach to Unsupervised Machine Translation (P19-1)
Copied to clipboard
| Challenge: | a recent research line has managed to train both unsupervised and unsupervised machine translation systems using monolingual corpora only. |
| Approach: | They propose to use monolingual corpora to train both unsupervised and unsupervised machine translation systems. |
| Outcome: | The proposed system achieves 22.5 BLEU points in English-to-German WMT 2014, 5.5 points more than the previous best unsupervised system, and 0.5 points more in the (supervised) shared task winner back in 2014. |
KoBE: Knowledge-Based Machine Translation Evaluation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for machine translation evaluation do not require reference translations. |
| Approach: | They propose a method for machine translation evaluation which does not require reference translations. |
| Outcome: | The proposed method achieves highest correlation with human judgements on 9 out of 18 language pairs from the WMT19 benchmark for evaluation without references. |
Rethinking Round-Trip Translation for Machine Translation Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Automatic evaluation methods for translation often require model training and the availability of parallel corpora limits their applicability to low-resource settings. |
| Approach: | They revisit the statistical machine translation technique and use it to improve translation quality. |
| Outcome: | The proposed method improves translation quality estimation models and identifies adversarial competitors in shared tasks via cross-system verification. |
English-Basque Statistical and Neural Machine Translation (L18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) requires large training corpora, which is problematic for low-resource languages. |
| Approach: | They propose to use an open-domain and an IT-domain corpora to train machine translations in English-Basque. |
| Outcome: | The proposed systems outperform OpenNMT, Moses SMT and Google Translate in English-Basque translation. |
NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) enables several similarity measures to estimate the probability of translations. |
| Approach: | They propose to rank the similarity of short text segments using translation-based similarity measures . they use the NMTScore library to analyze translation-related similarity . |
| Outcome: | The proposed measures show a relatively high correlation to human judgments when used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages. |
Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite (D19-1)
Copied to clipboard
| Challenge: | Currently, machine translation is performed at the level of individual sentences, in isolation from the rest of the document. |
| Approach: | They propose a dataset that can be used as a test suite for pronoun translation . they propose an evaluation measure to differentiate good and bad pronounce translations . |
| Outcome: | The proposed dataset can be used as a test suite for pronoun translation in English . it covers multiple source languages and different pronouner errors drawn from real system translations . |