The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents (2026.acl-long)
Copied to clipboard
| Challenge: | a fundamental pillar of trustworthiness is calibration, which refers to an agent’s ability to express confidence that reliably reflects its actual performance. |
| Approach: | They propose a reinforcement learning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. |
| Outcome: | The proposed framework improves calibration across tool types and shows that trained agents achieve superior calibration and exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. |
Similar Papers
Uncertainty Calibration for Tool-Using Language Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language agents are increasingly used to perform tasks and interact with a variety of external tools to achieve specific, goal-oriented objectives. |
| Approach: | They propose a tool calibration tool called ProbeCal which recalibrates the internal probabilities of tool-using language agents to better reflect the actual effectiveness of tool. |
| Outcome: | The proposed model significantly improves off-the-shelf language models in tool-using applications. |
Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in decision-making tasks where accuracy and reliable confidence estimates are essential. |
| Approach: | They propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. |
| Outcome: | The proposed model preserves RLVR’s accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a profound impact on a wide range of applications. |
| Approach: | They propose a framework to mitigate the tool-abuse behavior of Large Language Models and propose SMARTCAL to mitigate this issue. |
| Outcome: | The proposed framework improves the performance of LLMs on three datasets with two mainstream tool-use frameworks and shows an 8.6% increase in QA performance and 21.6 percent lower expected calibration error (ECE) than existing methods. |
Demystifying Uncertainty in LLMs: Active Calibration between Concepts and Human Evaluations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing static strategies for mitigating hallucinations do not explicitly model the information gain from interacting with the external environment. |
| Approach: | They propose a calibration-driven interactive learning strategy that selects clarification queries by optimizing calibration error. |
| Outcome: | The proposed method provides theoretical guarantees and empirical gains for reliability. |
ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration (2026.acl-long)
Copied to clipboard
| Challenge: | Existing training frameworks for Large Language Models (LLMs) focus on answers’ accuracy, overlooking specific alignment for behavior patterns. |
| Approach: | They propose a training framework for calibrating agent’s tool-use behavior through two synergistic perspectives: self-evolving data flywheel and behavior calibration training. |
| Outcome: | The proposed framework improves the accuracy, efficiency, reasoning conciseness, and tool execution accuracy of large language models. |
Learning to Use Tools via Cooperative and Interactive Agents (2024.findings-emnlp)
Copied to clipboard
Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Pengjie Ren, Suzan Verberne, Zhaochun Ren
| Challenge: | Existing methods for large language models (LLMs) use one agent to iterate and execute tools, but they suffer from performance degradation when addressing practical tasks. |
| Approach: | They propose a tool learning framework that coordinates three specialized agents for tool selection, tool execution, and action calibration separately. |
| Outcome: | The proposed framework outperforms baseline models on three datasets with 14% higher success rate. |
Thesis Proposal: When Does an Agent Know It Is Lost? Confidence Trajectory Analysis for Tool-Using LLMs (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing uncertainty quantification methods treat each step in isolation, ignoring how confidence evolves and compounds across a full task trajectory. |
| Approach: | They propose a framework for trajectory-level confidence analysis in the tool-use agent setting. |
| Outcome: | The proposed framework will expose early warning signals for agent failure and offer interpretable diagnostic tools for understanding when and why LLM agents lose confidence. |
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |