Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models have limited truthfulness and tendency toward overconfidence constrain reliability in factual tasks. |
| Approach: | They propose an efficient method that leverages semantic information encoded in LLMs to quantify uncertainty. |
| Outcome: | The proposed method achieves comparable performance to baselines while significantly reducing computational overhead. |
Similar Papers
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for uncertainty quantification in large language models rely on indirect signals, such as entropy across sampled generations, which can be difficult to interpret and do not fully leverage the model’s ability to assess its own uncertainty. |
| Approach: | They propose a method that groups sampled generations into semantically distinct clusters and uses the probability assigned by the LLM to each option as a confidence estimate. |
| Outcome: | The proposed method outperforms baseline methods and achieves competitive performance with as few as two additional samples. |
Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models (2025.naacl-long)
Copied to clipboard
Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). |
| Approach: | They propose to use a well-established method for text generation to extract token embeddings from multiple layers of LLMs and compute MD scores for each token. |
| Outcome: | The proposed method improves on existing methods and provides accurate and computationally efficient uncertainty scores for sequence-level selective generation and claim-level fact-checking tasks. |
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate long one-sentence responses that are less effective because they overlook two crucial factors: intra-cluster similarity and inter-c cluster similarity. |
| Approach: | They propose a method that generalizes semantic entropy and uses token probabilities to quantify uncertainty in large language models. |
| Outcome: | The proposed method can be extended to white-box settings by incorporating token probabilities. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |
Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on calibration and uncertainty quantification focuses on individual models, overlooking the potential of model diversity. |
| Approach: | They propose a method that uses Jensen-Shannon Divergence to identify and aggregate well-calibrated subsets of large language models (LLMs) to improve calibration. |
| Outcome: | The proposed method improves accuracy on binary prediction tasks compared to single-model and naive ensemble baselines. |
Uncertainty Quantification for Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce hallucinations, which undermine user trust and reliability. |
| Approach: | This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks. |
| Outcome: | The proposed framework provides tools for communicating the reliability of a model answer. |
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)
Copied to clipboard
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
| Challenge: | Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable. |
| Approach: | They propose to shift attention to more relevant components at token- and sentence-levels for better UQ. |
| Outcome: | The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters. |
Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for scaling test-time computation rely on external models that introduce substantial computational overhead and fail to capture context-aware semantics. |
| Approach: | They propose a method that leverages the generator LLM’s internal hidden states for clustering, eliminating the need for external models. |
| Outcome: | The proposed method improves the computational efficiency of test-time scaling while maintaining or exceeding the performance of existing methods. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
Evidential Semantic Entropy for LLM Uncertainty Quantification (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantifying uncertainty in large language models do not account for the effects of the semantics of sampled answers. |
| Approach: | They propose to incorporate the semantics of sampled answers to estimate entropy by incorporating the semantic of sample answers into the estimation methods. |
| Outcome: | The proposed method significantly improves uncertainty quantification performance. |