Calibration of Pre-trained Transformers

By Shrey Desai, Greg Durrett. 2020

431 citations recorded by the source corpus.

Pre-trained Transformers are now ubiquitous in natural language processing, but despite their high end-task performance, little is known empirically about whether they are calibrated. Specifically, do these models' posterior probabilities provide an accurate empirical measure of how likely the model is to be correct on a given example? We focus on BERT and RoBERTa in this work, and analyze their calibration across three tasks: natural language inference, paraphrase detection, and commonsense reasoning. For each task, we consider in-domain as well as challenging out-of-domain settings, where models face more examples they should be uncertain about. We show that: (1) when used out-of-the-box, pre-trained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5x lower; (2) temperature scaling is effective at further reducing calibration error in-domain, and using label smoothing to deliberately increase empirical uncertainty helps calibrate posteriors out-of-domain.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “, 2017), in particular for BERT fine-tuning (Desai and Durrett, 2020).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “To our knowledge, no previous work has examined whether attention is a well-calibrated estimator of these properties, though previous work as explored the calibration of Transformer outputs from an interpretability perspective [18].”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “performance on extremely large models with hundreds of billions of parameters trained on extremely large datasets [7, 20, 37, 45, 100].”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, background): “This is not only very difﬁcult to achieve across prompts (Desai and Durrett, 2020), but also unnecessary for ranking, which only requires relative ordering, a major focus of the learning to rank ﬁeld (Liu, 2009).”
Teaching Models to Express Their Uncertainty in Words (2022, methodology): “To improve the calibration of pre-trained language models, Desai & Durrett (2020) use label smoothing to reduce overconﬁdence on out-of-domain data.”
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2021, background): “, 2020) for CrowS-Pairs, however, recent work has suggested that pseudolikelihood-based scoring may be subject to model calibration issues (Desai and Durrett, 2020; Jiang et al., 2020).”
How Reliable are Model Diagnostics? (2021, background): “tree/master/official/nlp/bert The CrowS-Pairs diagnostic is expected to show higher variance than StereoSet for two reasons: (1) it is a smaller dataset (∼13 rd the size of StereoSetdev) with more categories, so results are more sensitive to changes in individual predictions; and (2) the pseudo-like”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, methodology): “…as part-of-speech tagging and named entity recognition (Jagannatha and Yu, 2020), natural language understanding tasks such as natural language inference, paraphrase detection, extractive question answering, and text clas-siﬁcation (Desai and Durrett, 2020; Kamath et al., 2020; Kong et al., 2020).”
Uncertainty Estimation for Language Reward Models (2022, background): “Moreover, language models’ uncertainty is unreliable out-of-domain [10, 21], and is poorly calibrated even in-domain for some tasks such as Q&A [20].”
Understanding the Effects of Iterative Prompting on Truthfulness (2024, methodology): “To gain more insight into this behavior, we analyzed the LLM’s calibration under iterative prompting [45].”
Prompting GPT-3 To Be Reliable (2022, methodology): “While various methods have been proposed to obtain conﬁdence scores and perform post-hoc calibration for language models (Jiang et al., 2021; Desai & Durrett, 2020; Ye & Durrett, 2022), they are mostly focused on classiﬁcation settings rather than free-form generation, which is more common for the…”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “First, a major obstacle to using PET in real-world applications is that we do not know a priori how well it performs for a given task; we therefore believe an important next step is to investigate methods for estimating performance without access to large test sets—for example, through model calibra”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, methodology): “Unfortunately, uncalibrated models are not uncommon in ML studies (Desai & Durrett, 2020; Guo et al., 2017), thus deterring potential operators (e.g., clinicians) from embracing and applying these tools in their daily practice.”
Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness (2020, background): “Desai and Durrett (2020) ﬁnd that the pre-trained Transformers RoBERTa (Liu et al., 2019) and BERT (Devlin et al., 2019) are well-calibrated in-domain on the tasks Natural Language Inference (NLI), paraphrase detection, and commonsense reasoning.”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, background): “Prior work on uncertainty in foundation models for NLP has largely focused on the calibration of classiﬁers (Jiang et al., 2021; Desai & Durrett, 2020) and text regressors (Glushkova et al., 2021; Wang et al., 2022).”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, background): “However, previous studies in NLP have predominantly addressed the associated uncertainty quantification challenges in the same vein as classification or regression methodologies [8, 18, 21, 48].”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “For each model, we minimize cross-entropy loss and tune softmax temperature2 on dev to alleviate overconfidence on unseen examples (Guo et al., 2017; Desai and Durrett, 2020).”