AskQE: Question Answering as Automatic Evaluation for Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. |
| Approach: | They propose a question generation and answering framework that detects critical MT errors and provides actionable feedback to help users decide whether to accept or reject MT outputs even without the knowledge of the target language. |
| Outcome: | The proposed framework has higher Kendall’s Tau correlation and decision accuracy with human ratings compared to other QE metrics. |
Similar Papers
Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the impact of feedback on human decision-making are limited as people are not equipped to assess the quality of AI predictions. |
| Approach: | They compare the quality of MT inputs and outputs with explicit and implicit feedbacks that directly give users an assessment of translation quality using error highlights and LLM explanations. |
| Outcome: | The proposed model improves decision accuracy and appropriate reliance by using error highlights and explanations, and by using backtranslation and question–answer tables. |
LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing MT evaluation frameworks fail to capture dialect- and culture-specific errors in diglossic languages. |
| Approach: | They propose a hierarchical error taxonomy for diagnosing MT errors through six linguistic levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics. |
| Outcome: | The proposed framework produces 6,113 labeled error spans across 3,495 unique erroneous sentences . it is language-agnostic and can be easily applied to or adapted for other languages. |
SimQA: Detecting Simultaneous MT Errors through Word-by-Word Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | a good SimulMT system will allow the downstream QA system to answer correctly as quickly as possible. |
| Approach: | They propose a word-by-word question answering evaluation task to evaluate if models translate salient elements of a question correctly. |
| Outcome: | a new evaluation task aims to show whether models translate salient elements of a question accurately and quickly . evaluators can reveal weaknesses in existing neural systems, hallucinating or omitting facts . human evaluation is too costly and slow to guide system development, authors say . |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
Towards more equitable question answering systems: How much more data do you need? (2021.acl-short)
Copied to clipboard
| Challenge: | Question answering datasets in English are relatively new, but lack of linguistic diversity in the field is a challenge. |
| Approach: | They propose to use translation and cross-lingual transfer to produce QA systems in multiple languages to improve their performance. |
| Outcome: | The proposed approaches take advantage of existing resources to produce QA systems in multiple languages. |
Evaluation of Question Answer Generation for Portuguese: Insights and Datasets (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Automatic question generation is an increasingly important task that can be applied in educational settings, data augmentation for question-answering (QA), and conversational systems. |
| Approach: | They adapt and apply QAG approaches to generate question-answer pairs given context and look into strategies for error filtering and their effects. |
| Outcome: | The proposed methods can generate question-answer pairs in Portuguese, a widely spoken language that is underrepresented in natural language processing research. |
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Generating multiple-choice questions (MCQG) for professional exams is challenging due to outdated knowledge, hallucination issues, and prompt sensitivity. |
| Approach: | They propose a framework for converting medical cases into high-quality USMLE-style questions using a self-refine-based framework. |
| Outcome: | The proposed framework improves human expert satisfaction regarding quality and difficulty of medical questions. |
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Evaluating Automatic Metrics with Incremental Machine Translation Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that neural metrics are more reliable than non-neural metrics. |
| Approach: | They propose to use commercial machine translations to evaluate machine translation metrics based on their preference for more recent outputs. |
| Outcome: | The proposed dataset confirms several previous findings, including the advantage of neural metrics over non-neural ones, and also explores the debated issue of how MT quality affects metric reliability. |