| Challenge: | Existing methods for posterior calibration have been used to correct poorly calibrated posterior probabilities. |
| Approach: | They propose a posterior calibration procedure that optimizes posterior probability distributions while minimizing calibration errors. |
| Outcome: | The proposed procedure reduces calibration error and improves performance on both objectives. |
Similar Papers
Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for posterior calibration return uncalibrated estimations of class posteriors, thus leading to poorer generalization. |
| Approach: | They propose an end-to-end trained calibrator that directly optimizes the objective while minimizing the difference between predicted and empirical posterior probabilities. |
| Outcome: | The proposed calibrator reduces calibration error and improves performance on benchmark NLP classification tasks. |
Calibration of Pre-trained Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained Transformers dominate benchmark tasks but use a large number of self-attention heads across many layers in a way that is difficult to unpack. |
| Approach: | They analyze pre-trained Transformer models' posterior probabilities to determine whether they are calibrated for three tasks: natural language inference, paraphrase detection, and commonsense reasoning. |
| Outcome: | The models are calibrated in-domain and out-of-domain, and their calibration error out-domain can be as much as 3.5x lower. |
Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression (2022.tacl-1)
Copied to clipboard
| Challenge: | State-of-the-art classification and regression models are often not well calibrated and can be inaccurate. |
| Approach: | They quantify calibration of pre- trained language models for text regression . they apply uncertainty estimates to augment training data in low-resource domains . |
| Outcome: | The proposed model calibrations improve performance and generalizability in low-resource settings. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
On the Impact of Calibration Data in Post-training Quantization and Pruning (2024.acl-long)
Copied to clipboard
| Challenge: | Quantization and pruning are the foundations of compression for large language models . however, no prior work has investigated how calibration data impacts performance of compression methods. |
| Approach: | They propose an empirical study on the effect of calibration data on LLM performance. |
| Outcome: | The proposed methods improve performance in a post-training setting. |
Making Pre-trained Language Models both Task-solvers and Self-calibrators (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work shows that pre-trained language models can be effective for high-stake applications, but they become overconfident in their wrong predictions. |
| Approach: | They propose to use extra data to train pre-trained language models to effectively utilize training samples to make them both task-solvers and self-calibrators. |
| Outcome: | The proposed method can be used in three downstream applications, including selective classification, adversarial defense, and model cascading. |
Self-calibration for Language Model Quantization and Pruning (2025.naacl-long)
Copied to clipboard
| Challenge: | Quantization and pruning are fundamental approaches for model compression, but they require large computational resources. |
| Approach: | They propose to use model calibration data to generate synthetic calibrations to improve model performance. |
| Outcome: | The proposed method outperforms other methods using real data in a post-training setting. |
Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates (2021.eacl-main)
Copied to clipboard
Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, Alexander Panchenko
| Challenge: | Annotating training data for sequence tagging of texts is usually very time-consuming . active learning can help to reduce the amount of annotation required to train a good model by multiple times . |
| Approach: | They are the first to thoroughly investigate active learning and transfer learning for natural language processing . they propose to combine active learning with active learning to improve model acquisition . |
| Outcome: | The proposed combination of active learning and Bayesian uncertainty estimation improves performance and reduces obstacles for applying it in practice. |
Bag of Tricks for In-Distribution Calibration of Pretrained Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models (PLMs) often predict over-confidently. |
| Approach: | They propose to use ensemble learning and data augmentation to improve confidence calibration for PLMs by combining calibration techniques with a trade-off between accuracy and classification. |
| Outcome: | The proposed calibration method improves classification accuracy and confidence in pre-trained language models by combining several calibration techniques. |
Beyond Fixed-Length Calibration for Post-Training Compression of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has demonstrated that the sequence length of calibration data plays a crucial role in the effectiveness of post-training compression methods. |
| Approach: | They propose a calibration technique that applies masking along the sequence axis to normalized hidden states. |
| Outcome: | The proposed method improves perplexity and zero-shot downstream tasks performance. |