| Challenge: | Existing methods for text summarization have been investigated, but there are still gaps between them and human professionals. |
| Approach: | They analyze 8 major sources of errors on 10 representative summarization models manually. |
| Outcome: | Aiming to gain more understanding of summarization systems with respect to their strengths and limitations on a fine-grained syntactic and semantic level, we use 8 major sources of errors on 10 representative summarizing models. |
Similar Papers
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (P19-1)
Copied to clipboard
| Challenge: | Recent years have seen success in the use of deep neural networks on text summarization, but there is no clear understanding of why they perform so well or how they might be improved. |
| Approach: | They propose to use different types of model architectures to improve extractive summarization systems. |
| Outcome: | The proposed framework achieves state-of-the-art on CNN/DailyMail by a large margin based on observations and analysis. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |
Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects - A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | scholarly attention has turned to the development of text summarization methods that are more closely tailored and controlled to align with specific objectives and user needs. |
| Approach: | They formalize a controllable text summarization task and categorize controllability attributes according to their shared characteristics and objectives. |
| Outcome: | The proposed method is tailored to meet the specific intent and needs of users. |
Extractive Summarization with Text Generator (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing extractive systems lack gold training signals, thereby hindering learning of extractive models. |
| Approach: | They propose to use text generators to train extractive summarizers by approximating outputs of abstractive summaries. |
| Outcome: | The proposed method can be used to train extractive summarizers without training . it is shown that the approximated summaries correlate positively with the auxiliary summary outputs. |
Content Selection in Deep Learning Models of Summarization (D18-1)
Copied to clipboard
| Challenge: | Using deep learning models, we find that word embedding does not improve performance over simpler models. |
| Approach: | They propose to use sentence embedding to perform content selection across multiple domains . they propose to propose two alternative models that use auto-regressive sentence extraction . |
| Outcome: | The proposed models improve performance across news, personal stories, meetings, and medical articles. |
Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization benchmarks overlap in time with pre-training corpora and fine-tuning datasets. |
| Approach: | They propose a temporal generalization benchmark that contains data samples from 2010 to 2022 to understand the temporal ability of abstractive summarization models. |
| Outcome: | The proposed benchmark analyzes data samples from 2010 to 2022 to understand the temporal generalization ability of abstractive summarization models. |
Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extractive summarization lack coherence, despite improvements . a human-annotated dataset is used to improve coherency of extractive summary . |
| Approach: | They propose to use human-annotated datasets to create coherent extractive summaries . they use supervised fine-tuning and natural language user feedback to enhance coherence . |
| Outcome: | The proposed dataset shows that LLMs can produce coherent summaries with human feedback. |
Neural Text Summarization: A Critical Evaluation (D19-1)
Copied to clipboard
| Challenge: | Current approaches to text summarization use advanced attention and copying mechanisms, multi-task and multi-reward training techniques. |
| Approach: | They evaluate datasets, evaluation metrics, and models for text summarization . they highlight three primary shortcomings: 1) datasets leave task underconstrained; 2) models overfit layout biases . |
| Outcome: | The current evaluation protocol is weakly correlated with human judgment and does not account for factual correctness. |
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |