A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy (2021.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text summarization systems are expensive and time-consuming. |
| Approach: | They propose a training-free and reference-free summarization evaluation metric that incorporates a centrality-weighted relevance score and a self-referenced redundancy score. |
| Outcome: | The proposed evaluation metric outperforms existing methods on multi-document and single-document summarization evaluation. |
Similar Papers
Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for summarization are reference-based and correlate poorly with relevance . fluency, faithfulness, coherence and relevance are all measures of human evaluation . |
| Approach: | They propose a reference-free metric that correlates well with human evaluated relevance . n-gram importance weighting is used to weight a summary's importance . |
| Outcome: | The proposed metric can be used along reference-based metrics to improve their robustness in low quality reference settings. |
Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for summarization use human annotations as reference. |
| Approach: | They propose a new automatic reference-free evaluation metric that compares semantic distribution between source document and summary by pretrained language models and considers summary compression ratio. |
| Outcome: | The proposed metric is more consistent with human evaluation in terms of coherence, consistency, relevance and fluency. |
Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document summarization consider the informativeness of the assessed summary and require human-generated references for each test summary. |
| Approach: | They propose to evaluate summary qualities without reference summaries by unsupervised contrastive learning. |
| Outcome: | The proposed method outperforms other evaluation metrics even without reference summaries. |
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for document summarization require human annotations and annotations. |
| Approach: | They propose a method which measures the quality of a summary by measuring its semantic similarity with a pseudo reference summary, using contextualized embeddings and soft token alignment techniques. |
| Outcome: | The proposed method correlates better with human ratings by 18- 39% compared to the state-of-the-art evaluation metrics. |
DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-based metrics are limited by their reliance on human input. |
| Approach: | They propose to adapt some reference-based metrics to assess system summary against human-written references. |
| Outcome: | The proposed model outperforms reference-based metrics on two datasets and is comparable to reference-free metrics. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
HighRES: Highlight-based Reference-less Evaluation of Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing methods for summarizing documents are inconsistent due to the difficulty of manual evaluation. |
| Approach: | They propose a method where summaries are evaluated by multiple annotators against the source document via manually highlighted salient content. |
| Outcome: | The proposed method improves inter-annotator agreement while highlighting differences among systems. |
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on summarization evaluation without a human-written reference summary have shown high correlations with human ratings. |
| Approach: | They propose to judge summary quality by learning preference rank from corrupted summaries . they use Bradley-Terry power ranking model to learn preference rank . |
| Outcome: | Experiments on several datasets show that the proposed model can produce scores highly correlated with human ratings. |
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (C18-1)
Copied to clipboard
| Challenge: | Existing summary evaluation methods rely on multiple model summaries to evaluate quality of summary outputs. |
| Approach: | They propose a new summary evaluation approach that does not require human model summaries . they exploit compositional capabilities of word embeddings to develop features . |
| Outcome: | The proposed metric replicates human-generated summarization scores on data from TAC 2008 and 2009 . the features are then used to train a learning model for predicting the summary content quality in the absence of gold models. |
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |