On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications. |
| Approach: | They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty. |
| Outcome: | The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. |
Similar Papers
Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods. |
| Approach: | They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets. |
| Outcome: | The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. |
Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue (2022.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2019; Lewis e t al, 2019) show that Bayesian summarization methods can generate high quality summaries but suffer from a couple of issues when inputs lie far from the training data distribution. |
| Approach: | They propose to extend state-of-the-art summarization models with Monte Carlo dropout and perform multiple stochastic forward passes to approximate Bayesian inference. |
| Outcome: | The proposed method outperforms deterministic summarization models on multiple benchmark datasets. |
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)
Copied to clipboard
| Challenge: | Modern summarization models generate fluent but often factually unreliable outputs. |
| Approach: | They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation. |
| Outcome: | The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses. |
What Have We Achieved on Text Summarization? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text summarization have been investigated, but there are still gaps between them and human professionals. |
| Approach: | They analyze 8 major sources of errors on 10 representative summarization models manually. |
| Outcome: | Aiming to gain more understanding of summarization systems with respect to their strengths and limitations on a fine-grained syntactic and semantic level, we use 8 major sources of errors on 10 representative summarizing models. |
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution’s Characteristics (2025.acl-short)
Copied to clipboard
| Challenge: | Existing methods for estimating confidence in text generation do not account for many valid answers in generation tasks. |
| Approach: | They propose task-agnostic confidence metrics that rely solely on model probabilities without the need for further fine-tuning or heuristics. |
| Outcome: | The proposed models improve the accuracy of BART and Flan-T5 on summarization, translation, and question answering datasets. |
Unsupervised Selective Rationalization with Noise Injection (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised selective rationalization produces rationales alongside predictions, but does not ensure that the rationale contains a plausible explanation for the prediction. |
| Approach: | They propose a technique that injects noise between a rationale generator and a predictor to limit generation of implausible rationales. |
| Outcome: | The proposed method achieves significant improvements in plausibility and task accuracy over the state-of-the-art models while maintaining or improving model faithfulness. |
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (P19-1)
Copied to clipboard
| Challenge: | Recent years have seen success in the use of deep neural networks on text summarization, but there is no clear understanding of why they perform so well or how they might be improved. |
| Approach: | They propose to use different types of model architectures to improve extractive summarization systems. |
| Outcome: | The proposed framework achieves state-of-the-art on CNN/DailyMail by a large margin based on observations and analysis. |
Content Selection in Deep Learning Models of Summarization (D18-1)
Copied to clipboard
| Challenge: | Using deep learning models, we find that word embedding does not improve performance over simpler models. |
| Approach: | They propose to use sentence embedding to perform content selection across multiple domains . they propose to propose two alternative models that use auto-regressive sentence extraction . |
| Outcome: | The proposed models improve performance across news, personal stories, meetings, and medical articles. |
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using low-resource languages, we assess the quality of uncertainty estimates from a wide array of approaches, but with more data. |
| Approach: | They train models on sub-sampled datasets in three different languages to assess the confidence of a neural classifier. |
| Outcome: | The proposed models train on sub-sampled datasets in three different languages and show that the quality of uncertainty estimates suffers with more data. |
A Closer Look at Data Bias in Neural Extractive Summarization Models (D19-54)
Copied to clipboard
| Challenge: | In this paper, we examine the generalization behaviour of summarization models . we propose several properties of datasets that matter for generalization . |
| Approach: | They propose several properties of datasets which matter for generalization of summarization models. |
| Outcome: | The proposed approach improves the state-of-the-art model by rethinking the model design process on a typical dataset. |