From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study investigated hallucinations in multi-document summarization tasks . but, it is unclear how challenges arising from handling multiple documents affect outputs . |
| Approach: | They investigate how hallucinations manifest in large language models when summarizing topic-specific information from a set of documents. |
| Outcome: | The proposed benchmarks show that the models generate more hallucinations than baselines . the results highlight the need for more effective approaches to mitigate hallucinosity in MDS . |
Similar Papers
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs (2025.naacl-short)
Copied to clipboard
Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, Amin Ahmad
| Challenge: | Existing evaluations of hallucinations in large language models suffer from a lack of diversity and recency in the LLM and LLM families considered. |
| Approach: | They propose a summarization hallucination benchmark that challenges models to disagree on hallucines . they use models to generate answers or summaries from textual input . |
| Outcome: | The proposed model combines the best of 10 modern LLMs with ground truth annotations. |
Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved summarization, but they still face a challenge of hallucination. |
| Approach: | They propose a taxonomy of errors to address the problem of hallucination in LLMs . they propose two prompt-based approaches for fine-grained error detection . |
| Outcome: | The proposed model outperforms existing metrics in identifying the novel "Contextual Inference" error type. |
Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate coherent text and follow instructions across diverse tasks, but a critical challenge in scaling LLM applications is hallucination, where the generated content lacks factual grounding or deviates from the intended discourse context. |
| Approach: | They use summarization as a representative task to evaluate LLMs' capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinos. |
| Outcome: | The proposed model distinguishes between factual and non-factual hallucinations, and their performance bottlenecks. |
Tutorial Proposal: Hallucination in Large Language Models (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | Grasping the intricacies of hallucination in LLMs can be daunting, especially for those new to the field. |
| Approach: | This tutorial aims to bridge the gap between the field and the field of hallucination . it will explore the key aspects of hallucinonation, including benchmarking, detection, and mitigation techniques . |
| Outcome: | This tutorial will explore the key aspects of hallucination in LLMs . it will also explore the specific constraints and shortcomings of current approaches . |
How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucination (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite LLMs becoming increasingly multilingual, most studies on detecting and quantifying LLM hallucination are English-centric . |
| Approach: | They train a multilingual hallucination detection model and conduct a large-scale study across 30 languages and 6 open-source LLM families. |
| Outcome: | The proposed model is based on an English-centric model and annotates gold data for five high-resource languages. |
The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive Remediations (2023.emnlp-main)
Copied to clipboard
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | Recent advances in Large Language Models have generated widespread acclaim, but hallucination has also emerged as a by-product. |
| Approach: | They propose a fine-grained discourse on profiling hallucination based on its degree, orientation, and category . they categorize hallucines into six types: acronym ambiguity, generated golem, virtual voice, geographic erratum, time wrap . |
| Outcome: | The proposed method categorizes hallucination into six types based on their degree, orientation, and category . |
On Large Language Models’ Hallucination with Regard to Known Facts (2024.naacl-long)
Copied to clipboard
Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, Jie Zhou
| Challenge: | Large language models are successful in answering factoid questions but are also prone to hallucination. |
| Approach: | They propose self-reporting to the model when faced with such limitations. |
| Outcome: | The proposed classifier can detect hallucinations with an 88% success rate and can be used to answer factoid questions with correct answer knowledge. |
ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent-proposed evaluation metrics for large language models have a preference-bias . however, such metrics often lack interpretability and only offer a single score . |
| Approach: | They propose a metric that leverages the power of large language models to perform two sub-tasks: decomposing summaries into atomic content units and validating them against the source document. |
| Outcome: | The proposed metric improves faithfulness scores on three summarization evaluation benchmarks by 3% compared to the next-best metric. |
Can We Trust AI Doctors? A Survey of Medical Hallucination in Large Language and Large Vision-Language Models (2025.findings-acl)
Copied to clipboard
Zhihong Zhu, Yunyan Zhang, Xianwei Zhuang, Fan Zhang, Zhongwei Wan, Yuyan Chen, QingqingLong QingqingLong, Yefeng Zheng, Xian Wu
| Challenge: | Hallucination is a critical challenge for large language models and large vision-language models (LVLMs) however, dedicated research on medical hallucinations remains unexplored. |
| Approach: | They provide a unified perspective on medical hallucination for both LLMs and LVLMs, and delve into its causes. |
| Outcome: | The proposed models have demonstrated impressive performance on a variety of medical benchmarks. |