QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | despite its high utility, there are limitations concerning manual QE data creation. |
| Approach: | They propose to generate a Korean-English QE dataset that is fully automatic . they find that the algorithm is more accurate and faster than manual QE . |
| Outcome: | The proposed datasets show that they scale up to 1.58M and 6.58M, respectively, and show that the results are significantly better when compared to the previous datasets. |
Similar Papers
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Self-Supervised Quality Estimation for Machine Translation (2021.emnlp-main)
Copied to clipboard
Yuanhang Zheng, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, Yang Liu
| Challenge: | Training QE models require massive parallel data with hand-crafted quality annotations, which are time-consuming and labor-intensive to obtain. |
| Approach: | They propose a self-supervised method to evaluate machine-translated sentences without references by recovering masked target words. |
| Outcome: | The proposed method outperforms previous unsupervised methods on several QE tasks in different language pairs and domains. |
Unsupervised Quality Estimation for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia
| Challenge: | Existing approaches require large amounts of expert annotated data, computation, and time for training. |
| Approach: | They propose an unsupervised approach to QE where no training is required . they use a dataset that enables work on both black-box and glass-box approaches . |
| Outcome: | The proposed approach rivals state-of-the-art supervised QE models in terms of correlation with human judgments of quality. |
deepQuest: A Framework for Neural-based Quality Estimation (C18-1)
Copied to clipboard
| Challenge: | Predicting Machine Translation (MT) quality has been limited to word and sentence-level prediction. |
| Approach: | They propose a framework that can generalize neural QE approaches to the level of documents. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on document-level quality estimates and is 40 times faster to train. |
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers (2021.acl-short)
Copied to clipboard
| Challenge: | Existing word-level quality estimation models require labelled data for each language pair and expensive maintenance. |
| Approach: | They propose to use multilingual QE models to generalise across languages . they propose to train models on other language pairs to predict word-level quality . |
| Outcome: | The proposed models generalise well across languages, making them more useful in real-world scenarios. |
EPOQUE: An English-Persian Quality Estimation Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing human labeled QE datasets are limited to limited language pairs . a small subset of the proposed dataset can improve its performance by 8% . |
| Approach: | They propose to use an English-Persian QE dataset with manually annotated direct assessment labels to evaluate translation quality estimation models. |
| Outcome: | The proposed dataset improves on two state-of-the-art QE models by 8% . the proposed dataset contains 1000 translated sentences from English to Persian . |
Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output (2021.emnlp-main)
Copied to clipboard
| Challenge: | Post-editing (PE) machine translation (MT) output can save time and reduce errors. |
| Approach: | They propose to use automatic word-level quality estimation to predict correctness of MT output to flag problematic output. |
| Outcome: | The proposed model is not good enough to support human translations, but is based on a visualization reflecting uncertainty of the model. |
Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on MT evaluation characterize quality of output with a single number . a recent advancement in MT technologies has enabled higher-quality, more nuanced translations . |
| Approach: | They propose a 1200-sentence MQM evaluation benchmark for English-Korean and a reference-free QE setup to evaluate the quality of the translations. |
| Outcome: | The proposed model outperforms the existing model in style and accuracy. |
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset (2022.lrec-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, André F. T. Martins
| Challenge: | Existing datasets for machine translation quality estimation and post-editing have several shortcomings. |
| Approach: | They propose a dataset for machine translation quality estimation and automatic post-editing . they report the performance of baseline systems trained on the MLQE-PE dataset . |
| Outcome: | The proposed dataset contains human labels for up to 10,000 translations per language pair. |