Evaluating the Factual Consistency of Abstractive Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | a weakly-supervised approach is needed to verify factual consistency . auxiliary span extraction tasks are useful for verifying factual consistent summaries . |
| Approach: | They propose a weakly-supervised approach for verifying factual consistency . they transfer the model to summaries generated by several neural models . |
| Outcome: | The proposed approach outperforms models trained with strong supervision on source documents and human evaluations. |
Similar Papers
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems have a lack of a defined definition for the task . factual consistency is a key factor in summarizing, but there are still deficiencies . a new study shows that summarized summarisation models achieve improved performance . |
| Approach: | They propose a filtered summarization dataset with improved factual consistency to address this problem . they argue that the dataset should become a valid benchmark for developing and evaluating summarizing systems . |
| Outcome: | The proposed model improves on a popular summarization dataset with improved factual consistency. |
Improving Factual Consistency of Abstractive Summarization via Question Answering (2021.acl-long)
Copied to clipboard
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, Bing Xiang
| Challenge: | Recent studies show that about 30% of summaries generated by neural text summarization suffer from fact fabrication. |
| Approach: | They propose an automatic evaluation metric to measure factual consistency and a learning algorithm that maximizes the metric during model training. |
| Outcome: | The proposed method improves factual consistency and overall quality of summarization models. |
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often produce inconsistent statements or false facts. |
| Approach: | They propose an efficient weak-supervised adversarial data augmentation approach to generate factual consistency datasets by backpropagating gradients on token embeddings. |
| Outcome: | The proposed model can make interpretable factual errors tracing on public datasets and is cost-effective. |
Entity-level Factual Consistency of Abstractive Text Summarization (2021.eacl-main)
Copied to clipboard
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
| Challenge: | Existing models exhibit entity hallucination, generating names of entities that are not present in the source document. |
| Approach: | They propose to use entity-level factual consistency to improve model quality . they propose to filter the training data to reduce entity hallucination problem . |
| Outcome: | The proposed model can reduce the entity hallucination problem by filtering the training data. |
A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the recent progress for summarization models in producing fluent summaries, they still encounter challenges when long sequences of generated texts and inputs (over thousands of words) need to be evaluated. |
| Approach: | They conduct a systematic analysis of factual-consistency evaluation systems across four long-document datasets and examine the relationship between sentence-level and summary-level model performance. |
| Outcome: | The proposed models can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection. |
Fine-grained Factual Consistency Assessment for Abstractive Summarization Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that around 30% of the summaries generated by abstractive summarization models contain factual errors. |
| Approach: | They propose a fine-grained two-stage Fact Consistency assessment framework for summarization models that uses fine-grain consistency reasoning to find subtle clues to identify whether a model-generated summary is consistent with the original document. |
| Outcome: | The proposed framework improves on the state-of-the-art models and distinguishes detailed differences better. |
Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking (2022.findings-naacl)
Copied to clipboard
| Challenge: | Abstractive summarization systems generate paraphrases, but they often contain information inconsistent with the source text. |
| Approach: | They propose to generate factually inconsistent summaries using source texts and reference summary with key information masked to train a factual consistency classifier. |
| Outcome: | The proposed method outperforms existing models and shows a competitive correlation with human judgments. |
SummaCoz: A Dataset for Improving the Interpretability of Factual Consistency Detection for Summarization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Summarization is an important application of Large Language Models. |
| Approach: | They integrate human-annotated and model-generated natural language explanations to elucidate how a summary deviates and becomes inconsistent with its source article. |
| Outcome: | The proposed model provides rationales for its judgments and improves its accuracy significantly. |
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate factual consistency in text summarization neglect the intrinsic cause of factual inconsistency or rely on auxiliary tasks. |
| Approach: | They propose a method to evaluate the factual consistency in text summarization via counterfactual estimation, which formulates the causal relationship between source document, generated summary, and the language prior. |
| Outcome: | The proposed metric improves correlation with human judgments and convenience of usage on three public abstractive text summarization datasets. |
Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent content when parametric knowledge conflicts with knowledge in the input document. |
| Approach: | They propose a method to enhance factual adaptiveness while achieving factual consistency on original datasets. |
| Outcome: | The proposed method improves factual adaptiveness while achieving factual consistency on original datasets. |