Bag of Tricks for In-Distribution Calibration of Pretrained Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models (PLMs) often predict over-confidently. |
| Approach: | They propose to use ensemble learning and data augmentation to improve confidence calibration for PLMs by combining calibration techniques with a trade-off between accuracy and classification. |
| Outcome: | The proposed calibration method improves classification accuracy and confidence in pre-trained language models by combining several calibration techniques. |
Similar Papers
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
Calibration of Pre-trained Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained Transformers dominate benchmark tasks but use a large number of self-attention heads across many layers in a way that is difficult to unpack. |
| Approach: | They analyze pre-trained Transformer models' posterior probabilities to determine whether they are calibrated for three tasks: natural language inference, paraphrase detection, and commonsense reasoning. |
| Outcome: | The models are calibrated in-domain and out-of-domain, and their calibration error out-domain can be as much as 3.5x lower. |
Making Pre-trained Language Models both Task-solvers and Self-calibrators (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work shows that pre-trained language models can be effective for high-stake applications, but they become overconfident in their wrong predictions. |
| Approach: | They propose to use extra data to train pre-trained language models to effectively utilize training samples to make them both task-solvers and self-calibrators. |
| Outcome: | The proposed method can be used in three downstream applications, including selective classification, adversarial defense, and model cascading. |
On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs . |
| Approach: | They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models . |
| Outcome: | The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks. |
When High Accuracy Hides Poor Calibration: Rethinking Confidence Evaluation in Transformer-Based Text Classification with Balanced Brier Score (2026.acl-long)
Copied to clipboard
Guilherme Fonseca, Gabriel Prenassi, Washington Cunha, Leonardo Chaves Dutra da Rocha, Marcos André Gonçalves
| Challenge: | Existing evidence for TC under fine-tuning is limited. |
| Approach: | They propose a calibration method that balances the contribution of correct and incorrect predictions within confidence bins. |
| Outcome: | The proposed calibration measures show that the models are overconfident even when miscalibrated . the proposed calibration methods challenge calibration assessment practices and provide a more reliable alternative for evaluating confidence quality in Transformer-based TC. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction? (2023.acl-short)
Copied to clipboard
| Challenge: | Model ensemble has been in widespread use for Grammatical Error Correction (GEC), boosting model performance. |
| Approach: | They propose to use model ensembles computed by pre-trained language models to improve model performance. |
| Outcome: | The proposed ensembles do not improve but get worse after the PLM-based ensemble. |
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)
Copied to clipboard
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
| Challenge: | Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work . |
| Approach: | They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data . |
| Outcome: | The proposed models can achieve competitive or better performance than BERT under comparable conditions. |