Adaptive Platt Scaling with Causal Interpretations for Self-Reflective Language Model Uncertainty Estimates (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the ability of large language models to self-monitor and ask for human intervention. |
| Approach: | They propose a formal analysis of LLM self-reflection for uncertainty estimation using domain adaptation theory. |
| Outcome: | The proposed method improves accuracy and human interpretation on reasoning tasks. |
Similar Papers
Demystifying Uncertainty in LLMs: Active Calibration between Concepts and Human Evaluations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing static strategies for mitigating hallucinations do not explicitly model the information gain from interacting with the external environment. |
| Approach: | They propose a calibration-driven interactive learning strategy that selects clarification queries by optimizing calibration error. |
| Outcome: | The proposed method provides theoretical guarantees and empirical gains for reliability. |
SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to elicit confidence from large language models are limited to binary or inaccurate group-level confidence estimates. |
| Approach: | They propose a training framework that teaches LLMs to express more fine-grained confidence estimates. |
| Outcome: | The proposed training framework reduces the confidence calibration error and maintains the performance of the model. |
Can LLMs Learn Uncertainty on Their Own? Expressing Uncertainty Effectively in A Self-Training Manner (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excessive, random, and uninformative uncertainty rendering them unsuitable for decision-making in human-computer interactions. |
| Approach: | They propose an uncertainty-aware instruction tuning method that aligns LLMs’ perception with the probabilistic uncertainty of the generation. |
| Outcome: | The proposed method improves LLMs' performance by 45.2%, with reasonably good out-of-domain generalization capabilities. |
Calibrating Language Models with Adaptive Temperature Scaling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models' confidence scores are degraded after fine-tuning with reinforcement learning from human feedback. |
| Approach: | They propose a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. |
| Outcome: | Adaptive temperature scaling improves calibration by over 10% compared to prior methods . RLHF fine-tuning improves model accuracy, but degradation is not significant . |
Do Large Language Models Know What They Don’t Know? (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have vast knowledge that allows them to excel in various NLP tasks. |
| Approach: | They propose an automated method to detect uncertainty in the responses of large language models and a dataset to measure their self-knowledge. |
| Outcome: | The proposed method detects uncertainty in the responses of large language models and provides a novel measure of their self-knowledge. |
A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses. |
| Approach: | They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks . |
| Outcome: | The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem . |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
Mechanistic Interpretability of Emotion Inference in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) show promising capabilities in predicting human emotions from text. |
| Approach: | They investigate how autoregressive LLMs infer emotions by focusing on appraisal theory . they show that emotion representations are functionally localized to specific regions in the model . |
| Outcome: | The proposed model is functionally localized to specific regions in the model, and the results align with theoretical and intuitive expectations. |
Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong capabilities, enabling concise, context-aware answers in question answering tasks. |
| Approach: | They propose a framework that provides valid uncertainty guarantees for LLMs . they also propose 'model-agnostic' uncertainty estimation method that maintains valid guarantees even under noise. |
| Outcome: | The proposed method provides valid uncertainty guarantees even under noise. |
Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to self-detection only retrospectively evaluate LLM-generated answers, leading to over-trust in incorrectly generated answers. |
| Approach: | They propose a self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers to mitigate the over-trust in LLM generated incorrect answers. |
| Outcome: | The proposed framework can be integrated with existing approaches for superior self-detection. |