A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses. |
| Approach: | They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks . |
| Outcome: | The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem . |
Similar Papers
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)
Copied to clipboard
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. |
| Approach: | They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python. |
| Outcome: | The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Knowledge Boundary of Large Language Models: A Survey (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) store vast amount of knowledge in their parameters, but they still have limitations in the memorization and utilization of certain knowledge. |
| Approach: | They propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. |
| Outcome: | The proposed definition of the LLM knowledge boundary and taxonomy categorizes knowledge into four distinct types . aims to offer a comprehensive overview, facilitate access to key issues, and inspire further advancements in LLM research. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decoding strategies affect the probability distribution underlying the output of a language model and can therefore affect both generation quality and uncertainty. |
| Approach: | They investigate the impact of decoding strategies on uncertainty estimation in large language models . |
| Outcome: | The proposed methods improve the uncertainty estimation of large language models by reducing repetition. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024.eacl-srw)
Copied to clipboard
| Challenge: | a survey examines the landscape of mathematical problem-solving techniques . large language models have proven to be potent assets in unraveling nuances of mathematical reasoning . |
| Approach: | They examine the evolution of Large Language Models (LLMs) for solving mathematical problems . they examine the spectrum of LLM-oriented techniques proposed for solving math problems - and their challenges . |
| Outcome: | The survey examines the spectrum of proposed LLM-oriented techniques in solving math problems. |
Tutorial Proposal: Hallucination in Large Language Models (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | Grasping the intricacies of hallucination in LLMs can be daunting, especially for those new to the field. |
| Approach: | This tutorial aims to bridge the gap between the field and the field of hallucination . it will explore the key aspects of hallucinonation, including benchmarking, detection, and mitigation techniques . |
| Outcome: | This tutorial will explore the key aspects of hallucination in LLMs . it will also explore the specific constraints and shortcomings of current approaches . |