Confidence Estimation for LLMs in Multi-turn Interactions

Challenge:	Despite recent progress, most prior work studies confidence in single-turn question answering.
Approach:	They propose a logit-based probe that measures confidence in multi-turn dialogues . they propose 'infoECE' and a "hinter-guesser" paradigm for generating controlled evaluations based on data .
Outcome:	The proposed framework is grounded in calibration and monotonicity of confidence as more information becomes available.

Similar Papers

Overview

Challenge

Approach

Outcome

Confidence Should Be Calibrated More Than One Turn Deep (2026.acl-long)

Copied to clipboard

Zhaohan Zhang, Chengzhengxu Li, Xiaoming Liu, Chao Shen, Ziquan Liu, Ioannis Patras

Machine Translation

Challenge:	Existing work on confidence estimation and calibration focuses on single-turn settings . existing work on multi-turn calibration ignores the risks and potential of multi-turned conversations .
Approach:	They propose a multi-turn calibration task that reframes calibration from a static property into a dynamic challenge central to reliable multi- turn conversations.
Outcome:	The proposed model minimizes ECE@T and leverages ConfChat to improve confidence . the proposed model preserves and even enhances model performance in multi-turn interactions.

A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)

Copied to clipboard

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, Iryna Gurevych

Language Models

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations.
Approach:	They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned.
Outcome:	The proposed methods can be used to assess the reliability of models and to calibrate them across tasks.

All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)

Copied to clipboard

Caiqi Zhang, Chang Shu, Ehsan Shareghi, Nigel Collier

Semantic Text Processing

Challenge:	Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks.
Approach:	They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting.
Outcome:	The proposed methods improve confidence estimation and performance on two downstream tasks.

Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference (2026.findings-eacl)

Copied to clipboard

Bo-Wei Chen, Chung-Chi Chen, An-Zi Yen

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs.
Approach:	They propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates.
Outcome:	The proposed approach reduces token usage by approximately 60% and improves cost efficiency on the Massive Multitask Language Understanding (MMLU) benchmark.

MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)

Copied to clipboard

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

Language Models

Challenge:	Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions.
Approach:	They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4.
Outcome:	The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information.

Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD (2025.emnlp-main)

Copied to clipboard

Bryan Chen Zhengyu Tan, Daniel Wai Kit Chin, Zhengyuan Liu, Nancy F. Chen, Roy Ka-Wei Lee

Responsible & Trustworthy NLP

Challenge:	Large Language Models (LLMs) can struggle to balance gullibility to misinformation and resistance to valid corrections in persuasive dialogues.
Approach:	They propose a framework evaluating multi-turn stance-change dynamics across dual dimensions: persuasion type and domain.
Outcome:	The proposed framework improves LLM-3.1-8B-Instruct accuracy under misleading persuasion in safety contexts from 4.21% to 76.54%.

Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions (2025.findings-acl)

Copied to clipboard

Yubo Li, Yidi Miao, Xueying Ding, Ramayya Krishnan, Rema Padman

Language Models

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but their deployment in high-stake domains requires consistent and coherent behavior across multiple rounds of user interaction.
Approach:	They propose a framework for evaluating and improving LLM response consistency, and introduce a benchmark dataset to evaluate LLM consistency.
Outcome:	The proposed framework improves response stability without sacrificing accuracy, and offers a practical path toward more dependable behavior in critical, real-world deployments.

Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024.acl-long)

Copied to clipboard

Abhishek Kumar, Robert Morabito, Sanzhar Umbet, Jad Kabbara, Ali Emami

Semantic Text Processing

Challenge:	Large Language Models (LLMs) are increasingly used in high-stakes areas such as healthcare, law, and education.
Approach:	They propose a concept of Confidence-Probability Alignment that connects an LLM’s internal confidence to the confidence conveyed in the model’s response when explicitly asked about its certainty.
Outcome:	The proposed model shows the strongest confidence-probability alignment across a wide range of tasks.

BotChat: Evaluating LLMs’ Capabilities of Having Multi-Turn Dialogues (2024.findings-naacl)

Copied to clipboard

Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, Kai Chen

Dialogue Systems & Conversational Agents

Challenge:	Modern Large Language Models (LLMs) facilitate high-quality, multi-turn dialogues with humans, but human-based evaluation of such a capability requires substantial manual effort.
Approach:	They propose to evaluate LLMs' ability to emulate human-like, multi-turn conversations using an LLM-centric approach.
Outcome:	The proposed model emulates human-like, multi-turn conversations using an LLM-centric approach.

MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models (2025.findings-acl)

Copied to clipboard

Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, Wenxuan Zhang, Kam-Fai Wong

Language Models

Challenge:	Existing studies on LLM confidence estimations in languages other than English have been limited to English.
Approach:	They propose to use question-related language to prompt LLMs to assess their confidence in large language models.
Outcome:	The proposed model improves on question-related language prompts for LS tasks, while English exhibits notable linguistic dominance in confidence estimations.