| Challenge: | Existing summarization models are limited in measuring the factual inconsistency of generated summaries. |
| Approach: | They propose a decoder overconfidence-regularizing objective as a hallucination risk measurement to better estimate the quality of generated summaries. |
| Outcome: | The proposed metric is reference-free and requires no training or modules . it records state-of-the-art correlation to human judgment on three sets of summary-quality annotations. |
Similar Papers
ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent-proposed evaluation metrics for large language models have a preference-bias . however, such metrics often lack interpretability and only offer a single score . |
| Approach: | They propose a metric that leverages the power of large language models to perform two sub-tasks: decomposing summaries into atomic content units and validating them against the source document. |
| Outcome: | The proposed metric improves faithfulness scores on three summarization evaluation benchmarks by 3% compared to the next-best metric. |
FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs (2025.naacl-short)
Copied to clipboard
Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, Amin Ahmad
| Challenge: | Existing evaluations of hallucinations in large language models suffer from a lack of diversity and recency in the LLM and LLM families considered. |
| Approach: | They propose a summarization hallucination benchmark that challenges models to disagree on hallucines . they use models to generate answers or summaries from textual input . |
| Outcome: | The proposed model combines the best of 10 modern LLMs with ground truth annotations. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate coherent text and follow instructions across diverse tasks, but a critical challenge in scaling LLM applications is hallucination, where the generated content lacks factual grounding or deviates from the intended discourse context. |
| Approach: | They use summarization as a representative task to evaluate LLMs' capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinos. |
| Outcome: | The proposed model distinguishes between factual and non-factual hallucinations, and their performance bottlenecks. |
From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study investigated hallucinations in multi-document summarization tasks . but, it is unclear how challenges arising from handling multiple documents affect outputs . |
| Approach: | They investigate how hallucinations manifest in large language models when summarizing topic-specific information from a set of documents. |
| Outcome: | The proposed benchmarks show that the models generate more hallucinations than baselines . the results highlight the need for more effective approaches to mitigate hallucinosity in MDS . |
FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis. |
| Approach: | They propose a fine-grained evaluator specifically tailored for the summarization task using large language models. |
| Outcome: | The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods. |
Mutual Information Alleviates Hallucinations in Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models exhibit the tendency to hallucinate, i.e., output content not supported by the source document. |
| Approach: | They propose a decoding strategy that optimizes for pointwise mutual information of source and target tokens when models exhibit uncertainty. |
| Outcome: | The proposed method decreases the probability of hallucinated tokens while maintaining the Rouge and BERT-S scores of top-performing decoding strategies. |
Reducing Quantity Hallucinations in Abstractive Summarization (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive summaries are subject to hallucination, but they are not very informative. |
| Approach: | They propose to use a beam-worth of abstractive summaries to up-rank summary that is not supported by the original text. |
| Outcome: | The proposed system up-ranks summaries whose quantity terms are supported by the original text without losing Recall, and shows higher Precision. |
Entity-level Factual Consistency of Abstractive Text Summarization (2021.eacl-main)
Copied to clipboard
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
| Challenge: | Existing models exhibit entity hallucination, generating names of entities that are not present in the source document. |
| Approach: | They propose to use entity-level factual consistency to improve model quality . they propose to filter the training data to reduce entity hallucination problem . |
| Outcome: | The proposed model can reduce the entity hallucination problem by filtering the training data. |