Papers by Markus Freitag
Finding Replicable Human Evaluations via Stable Ranking Probability (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that human evaluation is the best way to rank natural language generation systems . human raters can exhibit different behaviors when rating outputs, causing ranking to be unstable . stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Approach: | They propose to evaluate results through the lens of stability: stability is the degree to which a specific evaluation methodology produces the same system ranking when repeated. |
| Outcome: | The proposed model is based on a dataset of multi-segment translations rated by multiple professionals . human raters can exhibit different behaviors when rating NLG outputs, the study shows . |
Assessing Reference-Free Peer Evaluation for Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate machine translation output are based on comparing MT output to one or more reference translations. |
| Approach: | They propose to use probabilities given by a large, multilingual model as a reference-free metric. |
| Outcome: | The proposed model is robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities. |
Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing variants of Kendall's tau have weaknesses in handling of ties, and in some situations can even be gamed. |
| Approach: | They propose to use Kendall's tau to meta-evaluate machine translation evaluation metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties and a tie calibration procedure that automatically introduces ties into metric scores. |
| Outcome: | The proposed model gives metrics credit for correctly predicting ties and introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. |
On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Modern unsupervised machine translation systems reach reasonable translation quality under clean and controlled data conditions. |
| Approach: | They compare unsupervised and supervised machine translation systems of similar quality . they combine the benefits of both methods into a single system . |
| Outcome: | The proposed system improves adequacy and fluency as measured by human evaluators. |
A Natural Diet: Towards Improving Naturalness of Machine Translation Output (2022.findings-acl)
Copied to clipboard
| Challenge: | MT evaluation often focuses on accuracy and fluency without paying much attention to translation style. |
| Approach: | They propose a method for training machine translation systems to achieve a more natural style by contrasting training data according to the naturalness of the target side. |
| Outcome: | The proposed method achieves lexical richness on par with human translations, and is preferred by human experts when compared to baseline translations. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
Unsupervised Natural Language Generation with Denoising Autoencoders (D18-1)
Copied to clipboard
| Challenge: | Unsupervised approaches to generating text from structured data are costly to obtain and limited to a limited domain. |
| Approach: | They propose an unsupervised approach that learns its parameters without the slot pairs on target sequences only. |
| Outcome: | The proposed approach can generate sentences out of corrupted data without supervision . it can be used in question answering and dialog systems, the authors show . |
Prompting PaLM for Translation: Assessing Strategies and Performance (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models trained on multilingual but not parallel text exhibit remarkable ability to translate between languages. |
| Approach: | They investigate the pathways language model which has demonstrated the strongest machine translation performance among similarly-trained LLMs to date. |
| Outcome: | The pathways language model (PaLM) has demonstrated the strongest machine translation performance among similarly-trained LLMs to date. |
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics (2022.tacl-1)
Copied to clipboard
| Challenge: | Neural machine translations are ranked below human translations in professional evaluations . |
| Approach: | They apply minimum bayes risk decoding to optimize different metrics of translation quality . they show that model estimates and translation quality only vaguely correlate . |
| Outcome: | The proposed method improves human translations with different models and metric. |
BLEU might be Guilty but References are not Innocent (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using a method to collect references and compare their value with human evaluations, we show that multi-reference BLEU does not improve the correlation for high quality output. |
| Approach: | They propose a method to compare the quality of automated metrics by analyzing references and comparing them with human evaluations. |
| Outcome: | The proposed method improves correlation with all modern evaluation metrics including embedding-based methods. |
Enhancing Human Evaluation in Machine Translation with Comparative Judgement (2025.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. |
| Approach: | They evaluate three annotation setups to integrate comparative judgment into human annotation for machine translation. |
| Outcome: | The proposed approach improves inter-annotator agreement and stability of the annotations. |
MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation (2026.acl-long)
Copied to clipboard
| Challenge: | a critical component of machine translation model development is evaluating model quality. |
| Approach: | They propose a two-stage version of the current translation evaluation paradigm (MQM) they propose re-annotation, which uses raters to review and edit annotations . |
| Outcome: | The proposed method improves annotation quality by finding errors missed in the first pass. |
KoBE: Knowledge-Based Machine Translation Evaluation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for machine translation evaluation do not require reference translations. |
| Approach: | They propose a method for machine translation evaluation which does not require reference translations. |
| Outcome: | The proposed method achieves highest correlation with human judgements on 9 out of 18 language pairs from the WMT19 benchmark for evaluation without references. |
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021.tacl-1)
Copied to clipboard
| Challenge: | a large study of machine translation systems shows poor evaluation procedures can lead to erroneous conclusions. |
| Approach: | They propose an evaluation methodology grounded in explicit error analysis based on the Multidimensional Quality Metrics framework. |
| Outcome: | The proposed evaluation methodology outperforms crowd workers in two languages . it shows that human-based metrics outperformed crowd workers . |
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects (2025.findings-acl)
Copied to clipboard
Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, Markus Freitag
| Challenge: | In order to evaluate large language models (LLMs), it is important to collect benchmark datasets in order to assess their multilingual performance. |
| Approach: | They extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages/dialects. |
| Outcome: | The proposed dataset covers 55 languages and provides best-performing MT systems in all 55 languages. |
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in machine translation (MT) have shown that minimum bayes risk decoding can be a powerful alternative to beam search. |
| Approach: | They propose to use epsilon-sampling to prune away all tokens with a smaller probability mass. |
| Outcome: | The proposed method outperforms beam search decoding and other methods in four languages. |
Translationese as a Language in “Multilingual” NMT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work examines the impact of translationese in machine translation evaluation using the WMT evaluation campaign. |
| Approach: | They propose to use a sentence-level classifier to distinguish translationese from original target text to generate a machine translation model that can produce more natural outputs at test time. |
| Outcome: | The proposed model produces more natural outputs at test time, yielding gains in human evaluation scores on accuracy and fluency. |
LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback (2024.findings-naacl)
Copied to clipboard
Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, Markus Freitag
| Challenge: | Recent large language models (LLMs) are leveraging human feedback to improve their output quality. however, human feedback is costly to collect, especially at inference time when the model provides new, unseen input. |
| Approach: | They propose an inference-time optimization method to refine large language models' output based on fine-grained feedback to pinpoint defects and guide iterative refinement . |
| Outcome: | The proposed method consistently outperforms baseline approaches on three text generation tasks, including machine translation, long-form question answering, and topical summarization. |
Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work on translationese neglects important factors and conclusions are mostly correlational but not causal. |
| Approach: | They use a dataset where MT training data are also labeled with human translation directions to examine the impact of translationese on machine translation evaluation. |
| Outcome: | The proposed model learns in the same direction as human translation directions. |