Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output (2021.emnlp-main)
Copied to clipboard
| Challenge: | Post-editing (PE) machine translation (MT) output can save time and reduce errors. |
| Approach: | They propose to use automatic word-level quality estimation to predict correctness of MT output to flag problematic output. |
| Outcome: | The proposed model is not good enough to support human translations, but is based on a visualization reflecting uncertainty of the model. |
Similar Papers
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
Self-Supervised Quality Estimation for Machine Translation (2021.emnlp-main)
Copied to clipboard
Yuanhang Zheng, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, Yang Liu
| Challenge: | Training QE models require massive parallel data with hand-crafted quality annotations, which are time-consuming and labor-intensive to obtain. |
| Approach: | They propose a self-supervised method to evaluate machine-translated sentences without references by recovering masked target words. |
| Outcome: | The proposed method outperforms previous unsupervised methods on several QE tasks in different language pairs and domains. |
An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers (2021.acl-short)
Copied to clipboard
| Challenge: | Existing word-level quality estimation models require labelled data for each language pair and expensive maintenance. |
| Approach: | They propose to use multilingual QE models to generalise across languages . they propose to train models on other language pairs to predict word-level quality . |
| Outcome: | The proposed models generalise well across languages, making them more useful in real-world scenarios. |
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Measuring Uncertainty in Translation Quality Evaluation (TQE) (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing automated tools are not good enough to evaluate translation quality . existing tools are often accused of having low reliability and agreement . |
| Approach: | They propose to use a method to accurately estimate the confidence intervals depending on the sample size of the translated text. |
| Outcome: | The proposed method aims to estimate the confidence intervals (CITATION) depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. |
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern WQE techniques rely on expensive inference with large language models or ad-hoc training with large amounts of human-labeled data. |
| Approach: | They propose to use word-level quality estimation to identify translation errors from the inner workings of translation models to quantify the impact of human label variation on metric performance. |
| Outcome: | The proposed methods identify translation errors from the inner workings of translation models using human labels. |
MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset (2022.lrec-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, André F. T. Martins
| Challenge: | Existing datasets for machine translation quality estimation and post-editing have several shortcomings. |
| Approach: | They propose a dataset for machine translation quality estimation and automatic post-editing . they report the performance of baseline systems trained on the MLQE-PE dataset . |
| Outcome: | The proposed dataset contains human labels for up to 10,000 translations per language pair. |
deepQuest: A Framework for Neural-based Quality Estimation (C18-1)
Copied to clipboard
| Challenge: | Predicting Machine Translation (MT) quality has been limited to word and sentence-level prediction. |
| Approach: | They propose a framework that can generalize neural QE approaches to the level of documents. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on document-level quality estimates and is 40 times faster to train. |
Unsupervised Quality Estimation for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia
| Challenge: | Existing approaches require large amounts of expert annotated data, computation, and time for training. |
| Approach: | They propose an unsupervised approach to QE where no training is required . they use a dataset that enables work on both black-box and glass-box approaches . |
| Outcome: | The proposed approach rivals state-of-the-art supervised QE models in terms of correlation with human judgments of quality. |
QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | despite its high utility, there are limitations concerning manual QE data creation. |
| Approach: | They propose to generate a Korean-English QE dataset that is fully automatic . they find that the algorithm is more accurate and faster than manual QE . |
| Outcome: | The proposed datasets show that they scale up to 1.58M and 6.58M, respectively, and show that the results are significantly better when compared to the previous datasets. |