RISE: Leveraging Retrieval Techniques for Summarization Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Summarization evaluation approaches have relied on ROUGE for summarization, but they fall short of human evaluations. |
| Approach: | They propose a new approach to evaluate summaries by leveraging retrieval techniques . they use a dual-encoder retrieval setup to train a retrieval task . |
| Outcome: | The proposed method outperforms existing methods on two document summarization benchmarks and a long document summmarization test. |
Similar Papers
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (C18-1)
Copied to clipboard
| Challenge: | Existing summary evaluation methods rely on multiple model summaries to evaluate quality of summary outputs. |
| Approach: | They propose a new summary evaluation approach that does not require human model summaries . they exploit compositional capabilities of word embeddings to develop features . |
| Outcome: | The proposed metric replicates human-generated summarization scores on data from TAC 2008 and 2009 . the features are then used to train a learning model for predicting the summary content quality in the absence of gold models. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
HighRES: Highlight-based Reference-less Evaluation of Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing methods for summarizing documents are inconsistent due to the difficulty of manual evaluation. |
| Approach: | They propose a method where summaries are evaluated by multiple annotators against the source document via manually highlighted salient content. |
| Outcome: | The proposed method improves inter-annotator agreement while highlighting differences among systems. |
SummIt: Iterative Text Summarization via ChatGPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text summarization systems generate summaries in a single step, but are often inadequate due to the issue of hallucination and the lack of accuracy. |
| Approach: | They propose an iterative text summarization framework based on large language models like ChatGPT that refines the generated summary iterativly through self-evaluation and feedback. |
| Outcome: | The proposed framework refines the generated summary iteratively through self-evaluation and feedback, closely resembling the iteration humans undertake when drafting and revising summaries. |
Align then Summarize: Automatic Alignment Methods for Summarization Corpus Creation (2020.lrec-1)
Copied to clipboard
| Challenge: | Summarizing text is not a straightforward task. |
| Approach: | They propose to use automated transcriptions to generate reports from automatic transcriptions as a dataset for neural summarization. |
| Outcome: | The proposed model improves on publicmeetings corpus on a dataset of aligned public meetings. |
What Have We Achieved on Text Summarization? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text summarization have been investigated, but there are still gaps between them and human professionals. |
| Approach: | They analyze 8 major sources of errors on 10 representative summarization models manually. |
| Outcome: | Aiming to gain more understanding of summarization systems with respect to their strengths and limitations on a fine-grained syntactic and semantic level, we use 8 major sources of errors on 10 representative summarizing models. |
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (P19-1)
Copied to clipboard
| Challenge: | Recent years have seen success in the use of deep neural networks on text summarization, but there is no clear understanding of why they perform so well or how they might be improved. |
| Approach: | They propose to use different types of model architectures to improve extractive summarization systems. |
| Outcome: | The proposed framework achieves state-of-the-art on CNN/DailyMail by a large margin based on observations and analysis. |
Inducing Document Structure for Aspect-based Summarization (P19-1)
Copied to clipboard
| Challenge: | Abstractive summarization systems treat documents as unstructured and generate a single generic summary per document. |
| Approach: | They propose to incorporate document structure into automatic summarization systems . they induce latent document structure and abstractive summarizing objective . |
| Outcome: | The proposed model improves on topic-agnostic baselines and can produce abstractive and extractive aspect-based summaries. |
Beyond Generic Summarization: A Multi-faceted Hierarchical Summarization Corpus of Large Heterogeneous Data (L18-1)
Copied to clipboard
| Challenge: | Automated summarization has focused on ten to twenty documents, typically news articles, but could in theory analyze hundreds of documents from a wide range of sources and provide an overview to the interested reader. |
| Approach: | They propose a method for creating hierarchical summarization corpora from large, heterogeneous document collections by crowdsourcing relevant content and asking trained annotators to order the relevant information hierarchically. |
| Outcome: | The proposed method can be used to develop and evaluate hierarchical summarization systems. |