Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors (2023.emnlp-main)
Copied to clipboard
| Challenge: | a major challenge in the practical use of Machine Translation (MT) is that users lack information on translation quality to make informed decisions about how to rely on outputs. |
| Approach: | They evaluate quality estimation feedback in vivo with a human study in a medical setting. |
| Outcome: | The proposed method improves appropriate reliance on MT, but backtranslation helps detect harmful errors. |
Similar Papers
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the impact of feedback on human decision-making are limited as people are not equipped to assess the quality of AI predictions. |
| Approach: | They compare the quality of MT inputs and outputs with explicit and implicit feedbacks that directly give users an assessment of translation quality using error highlights and LLM explanations. |
| Outcome: | The proposed model improves decision accuracy and appropriate reliance by using error highlights and explanations, and by using backtranslation and question–answer tables. |
Unsupervised Quality Estimation for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia
| Challenge: | Existing approaches require large amounts of expert annotated data, computation, and time for training. |
| Approach: | They propose an unsupervised approach to QE where no training is required . they use a dataset that enables work on both black-box and glass-box approaches . |
| Outcome: | The proposed approach rivals state-of-the-art supervised QE models in terms of correlation with human judgments of quality. |
Self-Supervised Quality Estimation for Machine Translation (2021.emnlp-main)
Copied to clipboard
Yuanhang Zheng, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, Yang Liu
| Challenge: | Training QE models require massive parallel data with hand-crafted quality annotations, which are time-consuming and labor-intensive to obtain. |
| Approach: | They propose a self-supervised method to evaluate machine-translated sentences without references by recovering masked target words. |
| Outcome: | The proposed method outperforms previous unsupervised methods on several QE tasks in different language pairs and domains. |
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output (2021.emnlp-main)
Copied to clipboard
| Challenge: | Post-editing (PE) machine translation (MT) output can save time and reduce errors. |
| Approach: | They propose to use automatic word-level quality estimation to predict correctness of MT output to flag problematic output. |
| Outcome: | The proposed model is not good enough to support human translations, but is based on a visualization reflecting uncertainty of the model. |
Quality Estimation without Human-labeled Data (2021.eacl-main)
Copied to clipboard
Yi-Lin Tuan, Ahmed El-Kishky, Adithya Renduchintala, Vishrav Chaudhary, Francisco Guzmán, Lucia Specia
| Challenge: | Quality estimation aims to measure the quality of translated content without access to a reference translation. |
| Approach: | They propose a method that uses synthetic training data to train supervised quality estimation models. |
| Outcome: | The proposed model outperforms models trained on human-annotated data for sentence and word-level prediction. |
Measuring Uncertainty in Translation Quality Evaluation (TQE) (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing automated tools are not good enough to evaluate translation quality . existing tools are often accused of having low reliability and agreement . |
| Approach: | They propose to use a method to accurately estimate the confidence intervals depending on the sample size of the translated text. |
| Outcome: | The proposed method aims to estimate the confidence intervals (CITATION) depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. |
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023.eacl-main)
Copied to clipboard
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan
| Challenge: | State-of-the-art machine translation quality estimation systems have been achieving remarkable correlations with human judgements yet they require human annotations, which are expensive and computationally heavy. |
| Approach: | They propose a problem where one predicts automated metric scores without the reference. |
| Outcome: | The proposed model can estimate automated metrics at the sentence-level without the reference. |
Selecting Backtranslated Data from Multiple Sources for Improved Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | incorporating backtranslated data from different sources has led to improved results in machine translation (MT) |
| Approach: | They use a low-resource use-case and a high-resourced language pair to test different backtranslation scenarios and employ data selection to optimise the synthetic corpora. |
| Outcome: | The proposed method reduces the amount of data used while maintaining high-quality MT systems. |