| Challenge: | Critical evaluation decisions and parameters are routinely omitted, making most reports irreproducible . Thousands of papers use nonstandard evaluation packages with software defects that produce incorrect scores. |
| Approach: | a systematic review of over two thousand papers using a popular metric called ROUGE finds errors . critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible . a large number of ROUGEE model evaluation scores have been incorrectly computed . |
| Outcome: | a systematic review of over two thousand papers finds that ROUGE scores are incorrect . the metric is widely used in machine learning and is inconsistent with human evaluations . |
Similar Papers
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate text summarization tasks using ROUGE have been criticized for lack of semantic understanding. |
| Approach: | They propose a semantic-aware metric for extractive summarization task that is semantic-based . they use CNN/DailyMail dataset to study the new metric . |
| Outcome: | The proposed metric is semantic-aware and shows higher correlation with human judgement and yields a large number of disagreements with the original ROUGE metric. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for evaluating natural language generation has gained traction . lm judges have low intra-rater reliability in their assigned scores, making it difficult to measure how good their judgments actually are. |
| Approach: | They show that large language models align more closely with human preferences than n-grams . they quantify this variance and compare them to other NLG tasks and benchmarks based on the results . |
| Outcome: | The proposed models align more closely with human preferences than n-gram or embedding-based metrics. |
The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs (2025.emnlp-main)
Copied to clipboard
Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, Tomasz Jan Kajdanowicz
| Challenge: | Large language models (LLMs) have revolutionized natural language processing, but their tendency to hallucinate poses serious challenges for reliable deployment. |
| Approach: | They propose to use ROUGE to assess lexical overlap to determine accuracy of hallucination detection methods. |
| Outcome: | The proposed evaluation frameworks can rival complex methods, exposing a fundamental flaw in current evaluation practices. |
Are LLM-based Evaluators Confusing NLG Quality Criteria? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs confuse evaluation criteria, which reduces their reliability. |
| Approach: | They propose a hierarchical classification system for 11 common aspects with corresponding different evaluation criteria. |
| Outcome: | The proposed system is based on 11 common aspects with different evaluation criteria. |
In Benchmarks We Trust ... Or Not? (2025.emnlp-main)
Copied to clipboard
Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens, Andriy Kosar, Ehsan Lotfi, Nikolay Banar, Pieter Fivez, Luna De Bruyne, Walter Daelemans
| Challenge: | Existing benchmarks for Large Language Models (LLMs) are inadequate and lack a clear solution. |
| Approach: | They propose checklists to cover all aspects of benchmarking issues, both for benchmark creation and usage. |
| Outcome: | The proposed checklists cover all aspects of benchmarking issues, both for benchmark creation and usage. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |
Style Over Substance: Evaluation Biases for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Ranking the relative performance of large language models based on Elo ratings is gaining popularity . however, the extent to which humans and LLMs are capable evaluators remains uncertain . |
| Approach: | They propose to evaluate machine-generated text across multiple dimensions using the Elo rating system . they propose to use crowd-sourced and expert annotators to rank models based on Elo ratings . |
| Outcome: | The proposed method improves the quality of LLM-based evaluations, but there is no improvement in crowd-sourced evaluations. |