Representations of Fact, Fiction and Forecast in Large Language Models: Epistemics and Attitudes (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models estimate and calibrate confidence of large language models with verbalized uncertainty, but they lack a careful examination of the linguistic knowledge of uncertainty encoded in the latent space of LLMs. |
| Approach: | They draw on typological frameworks of epistemic expressions to evaluate LLMs’ knowledge of epistenetic modality, using controlled stories. |
| Outcome: | The proposed models generate expressions matching the strength of evidence and are not robust in generating epistemic expressions. |
Similar Papers
Defining Knowledge: Bridging Epistemology and Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on large language models (LLMs) define knowledge as a fact if it correctly completes a cloze sentence . but the predictions of semantically equivalent clozing sentences are inconsistent . |
| Approach: | They review standard definitions of knowledge in epistemology and formalize interpretations applicable to LLMs. |
| Outcome: | The authors compare the preferences of philosophers and computer scientists in terms of knowledge definitions and evaluation protocols for testing knowledge in accordance with the most relevant definitions. |
Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) tend to be unreliable on fact-based answers. |
| Approach: | They propose a framework for comparing LLMs' confidence over fact-based answers with hidden-state probes that are more reliable than hidden-status probes. |
| Outcome: | The proposed methods show that hidden-state probes provide the most reliable confidence estimates despite requiring access to weights and supervision data. |
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Known-unknown questions are characterized by high uncertainty due to the absence of definitive answers. |
| Approach: | They introduce a dataset with known-unknown questions and establish a categorization framework to clarify the origins of uncertainty in such queries. |
| Outcome: | The proposed model improved in distinguishing between known and unknown queries within open-ended question-answering scenarios. |
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. |
| Approach: | They argue that LLMs only parrot statistical patterns in training data and that language learning in LLM cannot inform human language learning. |
| Outcome: | The proposed model can generate grammatically correct, fluent text without requiring human intervention. |
Uncertainty Quantification for In-Context Learning of Large Language Models (2024.naacl-long)
Copied to clipboard
Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen
| Challenge: | Existing studies on in-context learning have focused on quantifying the uncertainty associated with the model's response, but they neglect the complexity of the LLM and the uniqueness of in-constitut learning. |
| Approach: | They propose a method to quantify the uncertainty associated with in-context learning and propose corresponding estimation method to quantify both types of uncertainties. |
| Outcome: | The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. |
Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite their unprecedented capabilities, large language models (LLMs) often output erroneous information, which may lead users to overly rely on their false output. |
| Approach: | They formalize faithful response uncertainty based on the gap between the model’s intrinsic confidence in the assertions it makes and the decisiveness by which they are conveyed. |
| Outcome: | The proposed model is poor at faithfully conveying uncertainty on knowledge-intensive questions. |
A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses. |
| Approach: | They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks . |
| Outcome: | The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem . |
Perceptions of Linguistic Uncertainty by Language Models and Humans (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that humans are well-attuned to the use of uncertainty expressions, exhibiting population-level agreement in mapping these expressions to numerical responses. |
| Approach: | They propose to map linguistic expressions of uncertainty to numerical responses by using a theory of mind approach to understand the uncertainty of another agent. |
| Outcome: | The proposed model can map expressions to probabilistic responses in a human-like manner, but different behavior depending on whether a statement is actually true or false. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
It’s Not What You Say, It’s How You Say It: Evaluating LLM Responses to Expressions of Belief (2026.acl-long)
Copied to clipboard
| Challenge: | a typology is grounded in four linguistically motivated dimensions: form, evidentiality, epistemic stance, and tone. |
| Approach: | They propose a typology to evaluate how different EoBs affect whether models follow context versus prior knowledge. |
| Outcome: | The proposed model systematically evaluates 16 LLMs that differ in architecture, scale, and training stages . human listeners subconsciously interpret the belief based on how it is expressed, i.e., its explicitness, tone, or contextual cues. |