PreSumm: Predicting Summarization Performance Without Summarizing (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in summarization models do not produce all documents in the same way, despite their inherent design principles and operational mechanisms. |
| Approach: | They propose a task where a system predicts summarization performance based solely on the source document. |
| Outcome: | The proposed task identifies documents that require manual summarization and improves dataset quality by filtering outliers and noisy documents. |
Similar Papers
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (C18-1)
Copied to clipboard
| Challenge: | Existing summary evaluation methods rely on multiple model summaries to evaluate quality of summary outputs. |
| Approach: | They propose a new summary evaluation approach that does not require human model summaries . they exploit compositional capabilities of word embeddings to develop features . |
| Outcome: | The proposed metric replicates human-generated summarization scores on data from TAC 2008 and 2009 . the features are then used to train a learning model for predicting the summary content quality in the absence of gold models. |
A Closer Look at Data Bias in Neural Extractive Summarization Models (D19-54)
Copied to clipboard
| Challenge: | In this paper, we examine the generalization behaviour of summarization models . we propose several properties of datasets that matter for generalization . |
| Approach: | They propose several properties of datasets which matter for generalization of summarization models. |
| Outcome: | The proposed approach improves the state-of-the-art model by rethinking the model design process on a typical dataset. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |
What Have We Achieved on Text Summarization? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text summarization have been investigated, but there are still gaps between them and human professionals. |
| Approach: | They analyze 8 major sources of errors on 10 representative summarization models manually. |
| Outcome: | Aiming to gain more understanding of summarization systems with respect to their strengths and limitations on a fine-grained syntactic and semantic level, we use 8 major sources of errors on 10 representative summarizing models. |
Abstractive Document Summarization with Summary-length Prediction (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing abstractive summarization models do not consider summarizing-specific information such as the target summary length. |
| Approach: | They propose a method for enabling a model to understand summarization-specific information by predicting the summary length in the encoder and generating a summary of the predicted length in fine-tuning. |
| Outcome: | The proposed method improves ROUGE scores on the WikiHow, NYT, and CNN/DM datasets. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |
CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for text summarization systems are limited to in-domain setting, where supervised pre-trained models are evaluated on the same dataset. |
| Approach: | They propose to use a cross-dataset evaluation approach to evaluate different summarization systems in a multi-domain setting. |
| Outcome: | The proposed model can be used to evaluate text summarization systems on different datasets. |
Which Information Matters? Dissecting Human-written Multi-document Summaries with Partial Information Decomposition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to characterize human-written summaries do not account for the nature of high-quality summary. |
| Approach: | They propose to characterize human-written summaries using partial information decomposition . they propose to decompose mutual information provided by all source documents into union, redundancy, synergy, and unique information . |
| Outcome: | The proposed approach decomposes the mutual information provided by all source documents into union, redundancy, synergy, and unique information. |
Content Selection in Deep Learning Models of Summarization (D18-1)
Copied to clipboard
| Challenge: | Using deep learning models, we find that word embedding does not improve performance over simpler models. |
| Approach: | They propose to use sentence embedding to perform content selection across multiple domains . they propose to propose two alternative models that use auto-regressive sentence extraction . |
| Outcome: | The proposed models improve performance across news, personal stories, meetings, and medical articles. |
Proceedings of the 2nd Workshop on New Frontiers in Summarization (D19-54)
Copied to clipboard
| Challenge: | EMNLP 2017 is a workshop on enhancing natural language processing's ability to produce concise, fluent summaries. |
| Approach: | the workshop provides a forum for cross-fertilization of ideas towards automatic summarization . four invited speakers will be present at the workshop . |
| Outcome: | the workshop aims to provide a forum for cross-fertilization of ideas towards automatic summarization. |