| Challenge: | ROUGE is one of the most widely used evaluation metrics for text summarization. |
| Approach: | They propose to use ROUGE to evaluate summaries based on lexical and semantic similarities. |
| Outcome: | The proposed method improves ROUGE's correlation with human judgments by exploiting lexical and semantic similarities. |
Similar Papers
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on automatic summary evaluation metrics focus on lexical similarity and require a reference summary which is expensive to obtain. |
| Approach: | They propose to use a weakly supervised summary evaluation approach without the presence of reference summaries to transform existing summarization datasets into corrupted reference summarizers. |
| Outcome: | The proposed method outperforms baselines and shows that it improves linguistic quality over all metrics. |
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate text summarization tasks using ROUGE have been criticized for lack of semantic understanding. |
| Approach: | They propose a semantic-aware metric for extractive summarization task that is semantic-based . they use CNN/DailyMail dataset to study the new metric . |
| Outcome: | The proposed metric is semantic-aware and shows higher correlation with human judgement and yields a large number of disagreements with the original ROUGE metric. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
An Anchor-Based Automatic Evaluation Metric for Document Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Existing reference-based evaluation metrics such as ROUGE have their own drawbacks. |
| Approach: | They propose a protocol for a reference-based automatic evaluation metric that requires the endorsement of source document. |
| Outcome: | The proposed metric is anchored on source document and has higher correlation with human judgments. |
Evaluating Multiple System Summary Lengths: A Case Study (D18-1)
Copied to clipboard
| Challenge: | Practical summarization systems are expected to produce summaries of varying lengths, per user needs. |
| Approach: | They propose to use ROUGE metric to evaluate system summaries of multiple lengths. |
| Outcome: | The evaluation protocol in question is competitive, the authors show . they found that the evaluation protocol is competitive with existing benchmarks. |
HOLMS: Alternative Summary Evaluation with Large Language Models (2020.coling-main)
Copied to clipboard
| Challenge: | Efficient document summarization requires evaluation measures that can rank a set of systems based on an average score and highlight which individual summary is better than another. |
| Approach: | They propose a hybrid evaluation measure for document summarization called HOLMS that combines both language models pre-trained on large corpora and lexical similarity measures. |
| Outcome: | The proposed measure outperforms ROUGE and BLEU on several extractive summarization datasets for both linguistic quality and pyramid scores. |
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization approaches based on Reinforcement Learning (RL) have been proposed to overcome classical likelihood maximization. |
| Approach: | They propose to use Reinforcement Learning to learn the model parameters through RL techniques to overcome classical likelihood maximization. |
| Outcome: | The proposed measures favor ROUGE with the additional property of not requiring reference summaries. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |