The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for summarization systems measure semantic overlap between a system summary and a human reference on word-string level. |
| Approach: | They propose to use distributed representations to evaluate system summary and human reference on word-string level. |
| Outcome: | The proposed representations outperform ROUGE on recent corpora but are less good on test data used in previous studies. |
Similar Papers
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on automatic summary evaluation metrics focus on lexical similarity and require a reference summary which is expensive to obtain. |
| Approach: | They propose to use a weakly supervised summary evaluation approach without the presence of reference summaries to transform existing summarization datasets into corrupted reference summarizers. |
| Outcome: | The proposed method outperforms baselines and shows that it improves linguistic quality over all metrics. |
An Anchor-Based Automatic Evaluation Metric for Document Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Existing reference-based evaluation metrics such as ROUGE have their own drawbacks. |
| Approach: | They propose a protocol for a reference-based automatic evaluation metric that requires the endorsement of source document. |
| Outcome: | The proposed metric is anchored on source document and has higher correlation with human judgments. |
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (C18-1)
Copied to clipboard
| Challenge: | Existing summary evaluation methods rely on multiple model summaries to evaluate quality of summary outputs. |
| Approach: | They propose a new summary evaluation approach that does not require human model summaries . they exploit compositional capabilities of word embeddings to develop features . |
| Outcome: | The proposed metric replicates human-generated summarization scores on data from TAC 2008 and 2009 . the features are then used to train a learning model for predicting the summary content quality in the absence of gold models. |
A Graph-theoretic Summary Evaluation for ROUGE (D18-1)
Copied to clipboard
| Challenge: | ROUGE is one of the most widely used evaluation metrics for text summarization. |
| Approach: | They propose to use ROUGE to evaluate summaries based on lexical and semantic similarities. |
| Outcome: | The proposed method improves ROUGE's correlation with human judgments by exploiting lexical and semantic similarities. |
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent generative language models have shown promise in abstractive summarization tasks. |
| Approach: | They propose to use Fr echet embedding distance and angular embeddable similarity to evaluate the performance of generative language models in abstractive summarization tasks. |
| Outcome: | The proposed metric shows close relation with human judgments and has overall better correlations with them. |
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate text summarization tasks using ROUGE have been criticized for lack of semantic understanding. |
| Approach: | They propose a semantic-aware metric for extractive summarization task that is semantic-based . they use CNN/DailyMail dataset to study the new metric . |
| Outcome: | The proposed metric is semantic-aware and shows higher correlation with human judgement and yields a large number of disagreements with the original ROUGE metric. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |