Papers with Calibration
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Model calibration is commonly used as a proxy for reliability, but low calibration error does not necessarily imply reliable decision rules. |
| Approach: | They investigate the calibration error gap in generative language models using the TOFU benchmark and attribution-based shortcut detection. |
| Outcome: | The proposed model calibrations achieve low calibration error compared to pretrained models and retain low calibration despite reduced accuracy on the forget split. |
Stop Measuring Calibration When Humans Disagree (2022.emnlp-main)
Copied to clipboard
| Challenge: | Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., predictive probabilities are a good indication of how likely a prediction is to be correct. |
| Approach: | They propose to measure calibration to human majority given inherent disagreements on tasks where humans inherently disagree about which class applies. |
| Outcome: | The proposed measures capture key statistical properties of human judgements including class frequency, ranking and entropy. |
Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve confidence calibration of pre-trained language models are still a mystery. |
| Approach: | They propose a method that leverages model explanations to make models less confident with non-inductive attributions. |
| Outcome: | The proposed method improves confidence calibration in all settings and reduces calibration errors when combined with temperature scaling. |
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. |
| Approach: | They propose a framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. |
| Outcome: | The proposed framework integrates response agreement and focal loss with binary cross-entropy to improve calibration from baselines. |
Self-Consistency Boosts Calibration for Math Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions for math reasoning tasks use semantic parsing or AST decoding, but performance can degrade dramatically even with slight changes to the questions. |
| Approach: | They propose three calibration methods based on self-consistency for math reasoning tasks. |
| Outcome: | The proposed methods bridge model confidence and accuracy better than existing methods based on p(True) or logit. |
LPC: A Logits and Parameter Calibration Framework for Continual Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solve catastrophic forgetting problem are varied . current approaches to learn continuous learning are based on replay-based methods . |
| Approach: | They propose to calibrate parameters and logits so that preserving old parameters and generalized learning on new concepts can be solved simultaneously. |
| Outcome: | The proposed model achieves state-of-the-art performance in all scenarios. |
Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language models (LLMs) have the capability to engage In-context Learning (ICL) however, this particular learning paradigm suffers from high instability stemming from factors such as input distribution, order and prompt formats. |
| Approach: | They propose to quantify the information gain obtained in prediction after observing a given example candidate and to sample those with maximum IG. |
| Outcome: | The proposed method can yield an average relative improvement of 14.3% across six classification tasks using three LLMs. |
Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025.emnlp-main)
Copied to clipboard
Yinuo Wang, Baiyang Wang, Robert Mercer, Frank Rudzicz, Sudipta Singha Roy, Pengjie Ren, Zhumin Chen, Xindi Wang
| Challenge: | achieving comprehensive trustworthiness in medical QA poses significant challenges due to complexity of healthcare data, critical nature of clinical scenarios, and multifaceted dimensions of trustworthy AI. |
| Approach: | They examine six key dimensions of trustworthiness in medical QA . they compare how each dimension is evaluated in existing LLM-based systems . |
| Outcome: | The findings show that large language models have improved patient safety and effectiveness . the models exhibit critical trust failures when deployed in clinical settings . |