Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Similar Papers
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)
Copied to clipboard
| Challenge: | Modern summarization models generate fluent but often factually unreliable outputs. |
| Approach: | They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation. |
| Outcome: | The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |
Evaluating Factuality in Cross-lingual Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summmarization. |
| Approach: | They propose to analyze cross-lingual factuality by collecting annotations and generated summaries from models at summary level and sentence level. |
| Outcome: | The proposed dataset shows that over 50% of generated summaries contain factual errors with different characteristics from monolingual summarization. |
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the progress of factual evaluation methods, they are limited in their opacity and lack the ability to assess the factuality of the summaries. |
| Approach: | They propose to use a meta-evaluation methodology to diagnose the fine-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets. |
| Outcome: | The proposed method diagnoses the strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets and searches for directions for further improvement by data augmentation. |
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems have a lack of a defined definition for the task . factual consistency is a key factor in summarizing, but there are still deficiencies . a new study shows that summarized summarisation models achieve improved performance . |
| Approach: | They propose a filtered summarization dataset with improved factual consistency to address this problem . they argue that the dataset should become a valid benchmark for developing and evaluating summarizing systems . |
| Outcome: | The proposed model improves on a popular summarization dataset with improved factual consistency. |
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often generate inconsistent summaries containing factual errors or fabricated content. |
| Approach: | They propose to generate representative examples of non-factual summaries through infilling language models and train a robust fact-correction model to post-edit them to improve factual consistency. |
| Outcome: | The proposed model outperforms previous methods in correcting factual errors on two popular summarization datasets. |
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on factual inconsistency in abstractive summarization addresses this problem. |
| Approach: | They propose a dataset with fine-grained factual error annotations named DIASUMFACT and an unsupervised model named ENDERANKER. |
| Outcome: | The proposed model performs on par with the state-of-the-art models while requiring fewer resources. |
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice. |
| Approach: | They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently. |
| Outcome: | The proposed models can detect errors and reduce them by reranking alternative summaries. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |