Factual Dialogue Summarization via Learning from Large Language Models

Challenge:	Existing models generate fluent and coherent summaries, but inconsistencies can be found in generated summary.
Approach:	They propose to use symbolic knowledge distillation to improve the factual consistency of smaller pretrained models for dialogue summarization.
Outcome:	The proposed model outperforms baseline models in BART, PEGASUS, and Flan-T5 in factual consistency and accuracy.

Similar Papers

Overview

Challenge

Approach

Outcome

Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models (2024.naacl-long)

Copied to clipboard

Shuaijie She, Shujian Huang, Xingyun Wang, Yanke Zhou, Jiajun Chen

Natural Language Interfaces

Challenge:	LLMs generate responses following user's instructions, which requires high dialogue comprehension ability.
Approach:	They propose to evaluate LLMs' dialogue comprehension ability using a dialogue summarization task to derive factual questions from the generated summaries and use them as a more flexible measurement of dialogue comprehension.
Outcome:	The proposed model reduces the error rate by 11% on the dialogue summarization task.

Evaluating the Factual Consistency of Abstractive Text Summarization (2020.emnlp-main)

Copied to clipboard

Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher

Information Extraction & Text Mining

Challenge:	a weakly-supervised approach is needed to verify factual consistency . auxiliary span extraction tasks are useful for verifying factual consistent summaries .
Approach:	They propose a weakly-supervised approach for verifying factual consistency . they transfer the model to summaries generated by several neural models .
Outcome:	The proposed approach outperforms models trained with strong supervision on source documents and human evaluations.

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)

Copied to clipboard

Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

Summarization

Challenge:	Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization.
Approach:	They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries.
Outcome:	The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets.

Multilingual Summarization with Factual Consistency Evaluation (2023.findings-acl)

Copied to clipboard

Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, Mirella Lapata

Text Generation

Challenge:	Abstractive summarization models generate factually inconsistent summaries, reducing their utility for real-world applications.
Approach:	They propose to use data filtering and controlled generation to detect hallucinations in machine generated summaries.
Outcome:	The proposed models detect factual inconsistencies in machine generated summaries, but they focus on English only.

TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models (2023.emnlp-main)

Copied to clipboard

Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, Idan Szpektor

Language Models

Challenge:	Existing methods for evaluating factual consistency are limited in their effectiveness.
Approach:	They propose a method for generating synthetic data by annotating diverse model-generated summaries using a LLM.
Outcome:	The proposed method outperforms state-of-the-art models and the LLM teacher on TRUE benchmarks.

SummaCoz: A Dataset for Improving the Interpretability of Factual Consistency Detection for Summarization (2024.findings-emnlp)

Copied to clipboard

Ge Luo, Weisi Fan, Miaoran Li, Guoruizhe Sun, Runlong Zhang, Chenyu Xu, Forrest Bao

Explainability & Interpretability in NLP

Challenge:	Summarization is an important application of Large Language Models.
Approach:	They integrate human-annotated and model-generated natural language explanations to elucidate how a summary deviates and becomes inconsistent with its source article.
Outcome:	The proposed model provides rationales for its judgments and improves its accuracy significantly.

Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking (2022.findings-naacl)

Copied to clipboard

Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, Kyomin Jung

Information Extraction & Text Mining

Challenge:	Abstractive summarization systems generate paraphrases, but they often contain information inconsistent with the source text.
Approach:	They propose to generate factually inconsistent summaries using source texts and reference summary with key information masked to train a factual consistency classifier.
Outcome:	The proposed method outperforms existing models and shows a competitive correlation with human judgments.

Evaluating the Factual Consistency of Large Language Models Through News Summarization (2023.findings-acl)

Copied to clipboard

Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, Colin Raffel

Information Extraction & Text Mining

Challenge:	Existing LLMs generally assign a higher score to factually consistent summaries than to factualally inconsistent summary.
Approach:	They propose a benchmark to measure whether large language models prefer factually consistent continuations of inputs.
Outcome:	The proposed benchmark compares the scores an LLM assigns to a factually consistent versus a inconsistent summary for an input news article.

SWING: Balancing Coverage and Faithfulness for Dialogue Summarization (2023.findings-eacl)

Copied to clipboard

Kung-Hsiang Huang, Siffi Singh, Xiaofei Ma, Wei Xiao, Feng Nan, Nicholas Dingwall, William Yang Wang, Kathleen McKeown

Dialogue Systems & Conversational Agents

Challenge:	Existing approaches to dialogue summarization rely on features of conversation data.
Approach:	They propose to use natural language inference models to improve coverage and faithfulness . they use fine-grained training signals to encourage model to generate missing content .
Outcome:	The proposed model achieves higher faithfulness and coverage while maintaining conciseness compared to prior methods.

CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021.emnlp-main)

Copied to clipboard

Shuyang Cao, Lu Wang

Semantic Text Processing

Challenge:	Existing methods for generating abstractive summarization are inconsistent and rely on heuristically created data for error handling.
Approach:	They propose a contrastive learning formulation that leverages both positive and negative summaries to train summarization systems that are better at distinguishing between them.
Outcome:	The proposed learning framework produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training.