RUBRIC-MQM : Span-Level LLM-as-judge in Machine Translation For High-End Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing LLMs are unable to match outputs due to their open-ended nature . |
| Approach: | They propose a meta-evaluation strategy PromptCUE to evaluate cutting-edge LAJ-MT models such as GEMBA-MQM and a rubric-style prompt tailored to the characteristics of LLMs. |
| Outcome: | The proposed model is able to predict scores or identify errors for individual sentences and is reliable in the real world. |
Similar Papers
LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing MT evaluation frameworks fail to capture dialect- and culture-specific errors in diglossic languages. |
| Approach: | They propose a hierarchical error taxonomy for diagnosing MT errors through six linguistic levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics. |
| Outcome: | The proposed framework produces 6,113 labeled error spans across 3,495 unique erroneous sentences . it is language-agnostic and can be easily applied to or adapted for other languages. |
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts (2024.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for the automated evaluation of natural language texts are based on a large language model (LLM) that fails to agree with human judges and is not fully validated by the human judges. |
| Approach: | They propose a large language model (LLM) that generates a distribution over potential responses to assess multiple dimensions of interest. |
| Outcome: | The proposed framework predicts human judges' assessment of user satisfaction on a scale of 1–4 with an RMS error 0.5, a 2 improvement over the uncalibrated baseline. |
MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment. |
| Approach: | They propose a framework that automatically post-edits the original translation based on each error, thereby filtering out non-impactful errors. |
| Outcome: | The proposed framework improves reliability and quality of error spans against GEMBA-MQM, across eight LLMs in both high- and low-resource languages. |
Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to evaluate long documents, but they are limited by context window limitations. |
| Approach: | They propose to use granularity-aligned prompting and Focus Sentence Prompting to improve evaluation. |
| Outcome: | a new study shows that long texts lead to fewer error spans and reduced system ranking accuracy. |
M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown their potential to deliver human-like judgments. |
| Approach: | They propose a systematic LLM-based multi-agent framework for advanced LLM as-a-judge MT evaluation that integrates dimension-specific results into a final evaluation judgment. |
| Outcome: | The proposed framework outperforms existing LLM-as-a-judge methods and competes with state-of-the-art automatic metrics even when powered by a suboptimal model like GPT-4o mini. |
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are expensive and time-consuming. |
| Approach: | They propose a framework that utilizes LLMs to generate synthetic evaluation datasets . they propose meta-correlation to measure alignment between metric rankings and human benchmarks based on synthetic data . |
| Outcome: | The proposed framework achieves meta-correlations exceeding 0.9 in multilingual QA and replaces human judgment with synthetic evaluation datasets. |
PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to improve the quality of machine-generated natural language content have been limited due to the large token usage required by complex evaluation prompts. |
| Approach: | They propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. |
| Outcome: | The proposed approach reduces token usage and costs by 2.37 compared with larger LLMs for downstream evaluation. |
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025.findings-emnlp)
Copied to clipboard
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
| Challenge: | Currently, traditional evaluation methods struggle to detect subtle translation errors. |
| Approach: | They propose to use a dataset of human evaluations for English–Russian translations created by professional linguists to enable consistent and rich annotation. |
| Outcome: | The proposed protocol allows expert assessments without time pressure to yield substantially different results from standard evaluations. |
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored. |
| Approach: | They show how well machine-translated benchmarks match human span annotations on translations . they also show how strongly translation errors explain accuracy drops on translated benchmarks - a gap that is not addressed yet . |
| Outcome: | The proposed model matches human-level translations with human-language annotations on translations, but translation errors are associated with accuracy drops even after controlling for English correctness and source-side anomalies. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |