| Challenge: | Recent years have seen several shifts in summarization research, including extractive models. |
| Approach: | They propose a pipeline method for applying GPT-3.5 to summarize user reviews . they propose three new metrics targeting faithfulness, factuality, and genericity . |
| Outcome: | The proposed methods perform well in opinion summarization, the authors show . they also show that standard evaluation metrics do not reflect this performance . |
Similar Papers
Few-Shot Learning for Opinion Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that abstractive summarization models fail to capture their essential properties due to the high cost of summary production. |
| Approach: | They propose a few-shot framework for abstractive opinion summarization that bootstraps the output of an unsupervised model. |
| Outcome: | The proposed framework outperforms extractive and abstractive methods on Amazon and Yelp datasets. |
Towards Opinion Summarization of Customer Reviews (P18-3)
Copied to clipboard
| Challenge: | Existing methods to summarize text are limited to small, homogeneous datasets . authors outline future directions to solve these problems . |
| Approach: | They propose to use neural networks to generate summaries of user-generated travel reviews . they aim to take into account shifting opinions over time and address these issues . |
| Outcome: | The proposed method will make it easier for users of review sites to make more informed decisions. |
OpinionDigest: A Simple Framework for Opinion Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Abstractive opinion summarization framework outperforms competitors' summarizing frameworks . extractive approaches produce well-formed text, but selecting the most popular opinions is challenging . |
| Approach: | They propose an abstractive opinion summarization framework that trains a Transformer model to reconstruct reviews from extracted opinions. |
| Outcome: | The proposed framework outperforms baselines on Yelp and shows promising customization capabilities. |
Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for opinion summarization are deficient in epitomizing extensive reviews and offering opinion summaries from various angles. |
| Approach: | They propose a supervised opinion summarization framework that takes sentiment orientation into account and trains the summarizer to learn from sub-optimal and optimal review subsets. |
| Outcome: | The proposed framework generates pros, cons, and verdict summaries from hundreds of input reviews. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised (D18-1)
Copied to clipboard
| Challenge: | Existing methods for opinion summarization are knowledge-lean and require light supervision. |
| Approach: | They propose a neural framework for opinion summarization from online product reviews which is knowledge-lean and only requires light supervision. |
| Outcome: | The proposed framework improves over baselines and shows that opinion summaries are preferred by human judges according to multiple criteria. |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets are limited to newswire text, which is a fraction of extant genres in general and on the Web. |
| Approach: | They present a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. |
| Outcome: | The proposed dataset of English summaries in 12 written and spoken genres is compared with human outputs and compared to untuned and prompt-based approaches. |
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for document summarization require human annotations and annotations. |
| Approach: | They propose a method which measures the quality of a summary by measuring its semantic similarity with a pseudo reference summary, using contextualized embeddings and soft token alignment techniques. |
| Outcome: | The proposed method correlates better with human ratings by 18- 39% compared to the state-of-the-art evaluation metrics. |