Papers by Alex Fabbri
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to control document controllable summarization lack abundant labeled data. |
| Approach: | They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks. |
| Outcome: | The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Generating EDU Extracts for Plan-Guided Summary Re-Ranking (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to generate summary candidates for re-ranking produce redundant, and often low quality, content. |
| Approach: | They propose a method to generate candidates for re-ranking that addresses these issues by grounding each abstract on its own unique content plan and creating distinct plan-guided abstracts using a model's top beam. |
| Outcome: | The proposed method outperforms baseline decoding methods on CNN, NYT, and Xsum and shows that prompting GPT-3 to follow EDU plans outperformed sampling-based methods by 1.05 points. |
Prompted Opinion Summarization with GPT-3.5 (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen several shifts in summarization research, including extractive models. |
| Approach: | They propose a pipeline method for applying GPT-3.5 to summarize user reviews . they propose three new metrics targeting faithfulness, factuality, and genericity . |
| Outcome: | The proposed methods perform well in opinion summarization, the authors show . they also show that standard evaluation metrics do not reflect this performance . |
Improving Factual Consistency in Summarization with Compression-Based Post-Editing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve factual consistency of summarization models fail to remove entity errors if a suitable input entity replacement is not available or insert erroneous content. |
| Approach: | They propose to remove extrinsic entity errors, or entities not in the source, to improve consistency while retaining the summary’s essential information and form. |
| Outcome: | The proposed model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and can be applied on top of another post-editor, improving accuracy by 38%. |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work suggests that the degree of hallucination depends on factual errors in training data. |
| Approach: | They propose a method to use training data to reduce hallucination by ensembling parameter variations in training data. |
| Outcome: | The proposed method improves on XSUM and CNN/DM datasets on human evaluations and factual metrics. |
Surfer100: Generating Surveys From Web Resources, Wikipedia-style (2022.lrec-1)
Copied to clipboard
Irene Li, Alex Fabbri, Rina Kawamura, Yixin Liu, Xiangru Tang, Jaesung Tae, Chang Shen, Sally Ma, Tomoe Mizutani, Dragomir Radev
| Challenge: | Recent work on Wikipedia page generation focuses on generating the initial leading paragraph of a page, while recent pretrained language models improve upon both extractive and abstractive steps of previous models. |
| Approach: | They propose a pretrained language model that can be combined to generate Wikipedia-style summaries with sections using 100 reference human-collected surveys. |
| Outcome: | The proposed approach is compared with existing methods with 100 human-collected surveys. |