How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains

Challenge:	Large Reasoning Models often struggle with confidence calibration, authors say . authors: accurate confidence scores are essential to build trustworthy systems .
Approach:	They propose a Reasoning Model Confidence estimation benchmark to assess LRM confidence . the benchmark is constructed from 347,496 reasoning traces from six popular LRMs .
Outcome:	The proposed benchmark compares ten different representation-based methods on a wide range of architectures.

Similar Papers

Overview

Challenge

Approach

Outcome

AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms? (2026.findings-acl)

Copied to clipboard

Henan Sun, Kaichi Yu, Yuyao Wang, Bowen Liu, Xunkai Li, Rong-Hua Li, Nuo Chen, Jia Li

Reasoning

Challenge:	Existing benchmarks for algorithmic reasoning fail to answer a critical question: do LRMs master algorithmic thinking? Empirical evaluations on leading LRM models reveal substantial performance heterogeneity, while models perform well on non-optimized tasks, accuracy drops sharply to around 49% on globally optimized algorithms.
Approach:	They propose an algorithm-centric benchmark that evaluates large reasoning models under an algorithmic paradigm.
Outcome:	Empirical evaluations on leading LRMs reveal substantial performance heterogeneity . models perform well on non-optimized tasks, accuracy drops sharply to around 49% .

All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)

Copied to clipboard

Caiqi Zhang, Chang Shu, Ehsan Shareghi, Nigel Collier

Semantic Text Processing

Challenge:	Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks.
Approach:	They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting.
Outcome:	The proposed methods improve confidence estimation and performance on two downstream tasks.

A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)

Copied to clipboard

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, Iryna Gurevych

Language Models

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations.
Approach:	They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned.
Outcome:	The proposed methods can be used to assess the reliability of models and to calibrate them across tasks.

Red Teaming Large Reasoning Models (2026.acl-long)

Copied to clipboard

Jiawei Chen, Yang Yang, Chao Yu, Yu Tian, Zhi Cao, Xue Yang, Linghao Li, Hang Su, Zhaoxia Yin

Responsible & Trustworthy NLP

Challenge:	Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, but they introduce safety and reliability risks, such as CoT-hijacking and prompt-induced inefficiencies.
Approach:	They propose a unified benchmark to assess the trustworthiness of Large Reasoning Models.
Outcome:	The proposed benchmark evaluates truthfulness, safety and efficiency on 26 models.

FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging (2025.acl-long)

Copied to clipboard

Zichen Tang, Haihong E, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, Qianhe Zheng

Reasoning

Challenge:	Compared to existing benchmarks, FinanceReasoning provides three key advancements: (1) credibility; (2) comprehensiveness; (3) numerical precision; (4) complexity; (5) complexity; and (6) complexity.
Approach:	They propose a benchmark to evaluate the reasoning capabilities of large reasoning models (LRMs) in financial numerical reasoning problems.
Outcome:	The proposed benchmark exceeds existing benchmarks in 67.8% of financial concepts and formulas and is credible, comprehensive, and challenging.

Safety in Large Reasoning Models: A Survey (2025.findings-emnlp)

Copied to clipboard

Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, Jiaheng Zhang, Bryan Hooi

Reasoning

Challenge:	Large Reasoning Models (LRMs) have a high level of advanced reasoning capabilities, but they are vulnerable and vulnerable.
Approach:	This paper presents the first comprehensive survey of Large Reasoning Models . it explores the new safety risks, attacks, and defense strategies specific to LRMs based on reasoning .
Outcome:	The proposed study examines the safety and security risks of large reasoning models.

Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning (2026.findings-acl)

Copied to clipboard

Zhenjiang Mao, Anirudhh Venkat, Artem Bisliouk, Sindhura Kumbakonam Subramanian, Akshat Kothiyal, Saithej Singhu, Ivan Ruchkin

Language Models

Challenge:	Existing confidence estimation methods reduce reasoning process to a single scalar score, ignoring how confidence evolves throughout generation.
Approach:	They propose to characterize the stepwise confidence signal using Signal Temporal Logic (STL) based on a discriminative STL mining procedure, they find temporal formulas that distinguish correct and incorrect responses.
Outcome:	The proposed method can distinguish between correct and incorrect reasoning signals.

Calibrating the Confidence of Large Language Models by Eliciting Fidelity (2024.emnlp-main)

Copied to clipboard

Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, Xipeng Qiu

Language Models

Challenge:	Large language models with RLHF and RLAIF have good alignment but exhibit overconfidence post-alignment.
Approach:	They propose a plug-and-play method to estimate the confidence of large language models.
Outcome:	The proposed method has shown good calibration performance on 6 RLHF-LMs on four MCQA datasets.

Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models’ Uncertainty? (2025.acl-short)

Copied to clipboard

Jiayu Liu, Qing Zong, Weiqi Wang, Yangqiu Song

Semantic Text Processing

Challenge:	Large language models (LLMs) are increasingly used in high-stakes domains, but their confidence is inconsistent in out-of-distribution scenarios.
Approach:	They define "marker confidence" as the observed accuracy when a model employs an epistemic marker.
Outcome:	The proposed model generalizes well within the same distribution, but its confidence is inconsistent in out-of-distribution scenarios.

Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference (2026.findings-eacl)

Copied to clipboard

Bo-Wei Chen, Chung-Chi Chen, An-Zi Yen

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs.
Approach:	They propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates.
Outcome:	The proposed approach reduces token usage by approximately 60% and improves cost efficiency on the Massive Multitask Language Understanding (MMLU) benchmark.