| Challenge: | Existing open-source frameworks for QE are based on complex ensemble systems, complicated architectures, or require not well-documented pretraining and fine-tuning of some components. |
| Approach: | They introduce OpenKiwi, a Pytorch-based framework for translation quality estimation. |
| Outcome: | The proposed framework performs state-of-the-art on word-level and sentence-level tasks and is near state-outperforming on sentence- and word-based tasks. |
Similar Papers
SpeechQE: Estimating the Quality of Direct Speech Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored. |
| Approach: | They propose a new quality estimation system based on cascaded and end-to-end architectures. |
| Outcome: | The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation. |
deepQuest: A Framework for Neural-based Quality Estimation (C18-1)
Copied to clipboard
| Challenge: | Predicting Machine Translation (MT) quality has been limited to word and sentence-level prediction. |
| Approach: | They propose a framework that can generalize neural QE approaches to the level of documents. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on document-level quality estimates and is 40 times faster to train. |
Assessing Quality Estimation Models for Sentence-Level Prediction (C18-1)
Copied to clipboard
| Challenge: | Using a relevant QE model is also very important in QE. |
| Approach: | They evaluate a wide range of advanced sentence-level Quality Estimation models including Support Vector Regression, Ride Regression and Bayesian Neural Networks. |
| Outcome: | The proposed models behave differently in evaluation settings depending on whether test data come from the same domain as the training data or not. |
deepQuest-py: Large and Distilled Models for Quality Estimation (2021.emnlp-demo)
Copied to clipboard
Fernando Alva-Manchego, Abiola Obamuyide, Amit Gajbhiye, Frédéric Blain, Marina Fomicheva, Lucia Specia
| Challenge: | Quality Estimation (QE) is a tool for machine translation that predicts how good translations are without comparing them to gold-standard references. |
| Approach: | They introduce a framework for training and evaluation of large and light-weight models for Quality Estimation (QE) they use pre-trained Transformers to train large and efficient QE models. |
| Outcome: | The framework provides access to state-of-the-art models based on pre-trained Transformers for sentence-level and word-level QE and a web interface for testing and visualising their predictions. |
Self-Supervised Quality Estimation for Machine Translation (2021.emnlp-main)
Copied to clipboard
Yuanhang Zheng, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, Yang Liu
| Challenge: | Training QE models require massive parallel data with hand-crafted quality annotations, which are time-consuming and labor-intensive to obtain. |
| Approach: | They propose a self-supervised method to evaluate machine-translated sentences without references by recovering masked target words. |
| Outcome: | The proposed method outperforms previous unsupervised methods on several QE tasks in different language pairs and domains. |
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for assessing translation quality rely on manual features and external knowledge. |
| Approach: | They propose to use a neural model without feature engineering to detect which parts in sentence pairs are most relevant for assessing quality. |
| Outcome: | The proposed model outperforms feature-based methods on a large human annotated dataset. |
TransQuest: Translation Quality Estimation with Cross-lingual Transformers (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in the field of sentence-level quality estimation (QE) are based on neural-based architectures that require resourceintensive training. |
| Approach: | They propose a framework for sentence-level quality estimation based on cross-lingual transformers and use it to implement and evaluate two different neural architectures. |
| Outcome: | The proposed framework outperforms open-source QE frameworks when trained on WMT datasets and is very competitive in transfer learning settings. |
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern WQE techniques rely on expensive inference with large language models or ad-hoc training with large amounts of human-labeled data. |
| Approach: | They propose to use word-level quality estimation to identify translation errors from the inner workings of translation models to quantify the impact of human label variation on metric performance. |
| Outcome: | The proposed methods identify translation errors from the inner workings of translation models using human labels. |