Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate long one-sentence responses that are less effective because they overlook two crucial factors: intra-cluster similarity and inter-c cluster similarity. |
| Approach: | They propose a method that generalizes semantic entropy and uses token probabilities to quantify uncertainty in large language models. |
| Outcome: | The proposed method can be extended to white-box settings by incorporating token probabilities. |
Similar Papers
Evidential Semantic Entropy for LLM Uncertainty Quantification (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantifying uncertainty in large language models do not account for the effects of the semantics of sampled answers. |
| Approach: | They propose to incorporate the semantics of sampled answers to estimate entropy by incorporating the semantic of sample answers into the estimation methods. |
| Outcome: | The proposed method significantly improves uncertainty quantification performance. |
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for uncertainty quantification in large language models rely on indirect signals, such as entropy across sampled generations, which can be difficult to interpret and do not fully leverage the model’s ability to assess its own uncertainty. |
| Approach: | They propose a method that groups sampled generations into semantically distinct clusters and uses the probability assigned by the LLM to each option as a confidence estimate. |
| Outcome: | The proposed method outperforms baseline methods and achieves competitive performance with as few as two additional samples. |
Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models have limited truthfulness and tendency toward overconfidence constrain reliability in factual tasks. |
| Approach: | They propose an efficient method that leverages semantic information encoded in LLMs to quantify uncertainty. |
| Outcome: | The proposed method achieves comparable performance to baselines while significantly reducing computational overhead. |
Uncertainty Quantification for Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce hallucinations, which undermine user trust and reliability. |
| Approach: | This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks. |
| Outcome: | The proposed framework provides tools for communicating the reliability of a model answer. |
MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty (2025.findings-naacl)
Copied to clipboard
| Challenge: | despite advances in large language models, they still produce false but incorrect responses. |
| Approach: | They propose a new benchmark for large language models that requires more than two unambiguous answers . they also assess 5 different uncertainty quantification methods in the presence of data uncertainty. |
| Outcome: | The proposed method fails in multi-answer question answering tasks compared to single-answered questions . entropy- and consistency-based methods effectively estimate model uncertainty, the authors show . |
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)
Copied to clipboard
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
| Challenge: | Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable. |
| Approach: | They propose to shift attention to more relevant components at token- and sentence-levels for better UQ. |
| Outcome: | The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters. |
Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty (2025.findings-emnlp)
Copied to clipboard
Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
| Challenge: | Existing methods to gauge model’s uncertainty through self-consistency in responses to the target query are misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same query when answering a knowledge-preserving perturbation of the query. |
| Approach: | They propose a method that uses multi-agent interaction to estimate black-box LLMs' uncertainty. |
| Outcome: | The proposed method outperforms existing self-consistency based methods and improves hallucination detection. |
Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy (2024.eacl-long)
Copied to clipboard
| Challenge: | Uncertainty estimation is an important diagnostic tool for statistical models. |
| Approach: | They propose to adapt similarity-sensitive Shannon entropy (S3E) for NMT by incorporating a concept borrowed from theoretical ecology. |
| Outcome: | The proposed framework improves quality estimation and named entity recall, and improves translation quality. |
Detecting LLM Hallucination Through Layer-wise Information Deficiency: Analysis of Ambiguous Prompts and Unanswerable Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate confident yet inaccurate responses, introducing significant risks for deployment in safety-critical domains. |
| Approach: | They propose a method to detect model hallucination by systematic analysis of information flow across model layers. |
| Outcome: | The proposed approach improves model reliability by immediately integrating with universal LLMs without additional training or architectural modifications. |
Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for scaling test-time computation rely on external models that introduce substantial computational overhead and fail to capture context-aware semantics. |
| Approach: | They propose a method that leverages the generator LLM’s internal hidden states for clustering, eliminating the need for external models. |
| Outcome: | The proposed method improves the computational efficiency of test-time scaling while maintaining or exceeding the performance of existing methods. |