Papers by Tom Kocmi
AI-Assisted Human Evaluation of Machine Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Annotation metrics are misaligned with the ideal measure of text quality and human evaluation remains the most accurate, reliable, and ultimate standard. |
| Approach: | They propose an annotation protocol that helps annotators mark erroneous parts of the translation and assign a final score. |
| Outcome: | The proposed protocol reduces the time per span annotation by half . the method reduces annotation budget by 25% with filtering of examples that the AI deems to be likely to be correct. |
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024.acl-long)
Copied to clipboard
| Challenge: | a decade ago a single metric, BLEU, governed progress in machine translation research. |
| Approach: | They investigate the "dynamic range" of a number of modern machine translation metrics to provide a collective understanding of differences in scores . they use a large dataset to discover deltas at which metrics achieve system-level differences that are meaningful to humans . |
| Outcome: | The proposed method is more stable than statistical p-values in regards to testset size. |
Estimating Machine Translation Difficulty (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the high-quality outputs, it is difficult to distinguish between state-of-the-art models and identify areas for future improvement. |
| Approach: | They propose a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches. |
| Outcome: | The proposed models outperform both heuristic-based methods and LLM-as-a-judge approaches, with sentinel-src achieving the best performance. |
SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window (2024.naacl-short)
Copied to clipboard
| Challenge: | prevailing evaluation methods for machine translation metrics are at the sentence-level . but there are many linguistic phenomena that cannot be translated without context . |
| Approach: | They propose a metric that leverages a moving window that slides over a document to feed it into a quality estimation model. |
| Outcome: | The proposed metric obtains significantly higher pairwise system accuracy than its sentence-level baseline. |
SumeCzech: Large Czech News-Based Summarization Dataset (L18-1)
Copied to clipboard
| Challenge: | Summarization of documents is a well-studied NLP task, but only a few datasets are available for Czech. |
| Approach: | They propose to use a Czech news-based summarization dataset to evaluate document summarizing . they propose a language-agnostic variant of the ROUGE metric to enable automatic evaluation . |
| Outcome: | The proposed dataset contains more than a million Czech news articles . the proposed approach is strong abstractive and language-agnostic . |
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References (2024.naacl-long)
Copied to clipboard
Tianyi Tang, Hongyuan Lu, Yuchen Jiang, Haoyang Huang, Dongdong Zhang, Xin Zhao, Tom Kocmi, Furu Wei
| Challenge: | Existing evaluation benchmarks with limited references may not accurately reflect the quality of the model’s hypotheses. |
| Approach: | They propose a method to enrich evaluation benchmarks by diversifying the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
| Outcome: | The proposed method can enhance evaluation benchmarks by diversifying the expression of reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (2023.eacl-main)
Copied to clipboard
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan
| Challenge: | State-of-the-art machine translation quality estimation systems have been achieving remarkable correlations with human judgements yet they require human annotations, which are expensive and computationally heavy. |
| Approach: | They propose a problem where one predicts automated metric scores without the reference. |
| Outcome: | The proposed model can estimate automated metrics at the sentence-level without the reference. |
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows that large language models (LLMs) perform poorly at segment level. |
| Approach: | They propose a new prompting method that emulates the commonly accepted human evaluation framework . they will release their code and scripts to facilitate the community . |
| Outcome: | The proposed method is based on the human evaluation framework MQM and produces explainable and reliable MT evaluations at both the system and segment level. |