How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

By Zhengbao Jiang, Jun Araki, Haibo Ding, Graham Neubig. 2020

594 citations recorded by the source corpus.

Abstract Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question, “How can we know when language models know, with confidence, the answer to a particular query?” We examine this question from the point of view of calibration, the property of a probabilistic model’s predicted probabilities actually being well correlated with the probabilities of correctness. We examine three strong generative models—T5, BART, and GPT-2—and study whether their probabilities on QA tasks are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb/lm-calibration.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “• Answer Paraphrasing Thesemethods start with an initial answer spaceZ, and then use paraphrasing to expand this answer space to broaden its coverage [51].”
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings (2022, methodology): “Inspired by calibration technique presented in Jiang et al. (2021), we also train calibrator as a regression model ( Calib R ) by annotating the heldout dataset on a continuous scale instead of categorical labels (positive and negative as done in Calib C).”
Active Retrieval Augmented Generation (2023, methodology): “Given the observation that large LMs tend to be well-calibrated and low probability/confidence often indicates a lack of knowledge (Jiang et al., 2021; Kadavath et al., 2022), we adopt an active retrieval strategy that only retrieves when LMs generate low-probability tokens.”
Large Language Models for Information Retrieval: A Survey (2023, unknown): “Since the probability can serve as an indicator of LLMs’ confidence during text generation [186, 187], a low probability for a generated term could suggest that LLMs require additional knowledge.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “Jiang et al. [212] studied Model Uncertainty, Input Uncertainty, and Input Statistics to comprehensively assess the confidence level of the model.”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “This can be problematic as state-of-the-art language models like BERT are overconﬁdent and poorly calibrated (Jiang et al., 2021).”
Uncertainty Estimation for Language Reward Models (2022, background): “Moreover, language models’ uncertainty is unreliable out-of-domain [10, 21], and is poorly calibrated even in-domain for some tasks such as Q&A [20].”
Cognitive Architectures for Language Agents (2023, background): “On the other hand, LLM development might be influenced or even shaped by the increased usage of reasoning toward complex decision making, and better support decision making by solving known issues such as over-confidence and miscalibration (Jiang et al., 2021; Braverman et al., 2020; Chen et al., 20”
Prompting GPT-3 To Be Reliable (2022, unknown): “While various methods have been proposed to obtain conﬁdence scores and perform post-hoc calibration for language models (Jiang et al., 2021; Desai & Durrett, 2020; Ye & Durrett, 2022), they are mostly focused on classiﬁcation settings rather than free-form generation, which is more common for the…”
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, unknown): “Recent research on LLM calibration shows that current LLMs’ probabilistic predictions correspond well with actual token occurrence frequencies, leading to well-calibrated predictions for specific tasks (Rae et al., 2021; Kadavath et al., 2022; Guo et al., 2017; Kadavath et al., 2022; Jiang et al., 2”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “…know a priori how well it performs for a given task; we therefore believe an important next step is to investigate methods for estimating performance without access to large test sets – for example, through model calibration (Desai and Durrett, 2020; Jiang et al., 2021) – in real-world settings.”
GPT-NER: Named Entity Recognition via Large Language Models (2023, background): “LLMs significantly suffer from the hallucination or overprediction issue (Braverman et al., 2020; Jiang et al., 2021; Zhao et al., 2021).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “Deep neural networks are known to generate uncalibrated uncertainty, leading to recent research looking into re-calibration techniques (Guo et al., 2017; Jiang et al., 2021).”
Large Language Models Are Not Robust Multiple Choice Selectors (2023, background): “We conjecture that an ideal alignment requires the model to be well-calibrated, i.e., the probabilistic predictions match up with frequencies of correctness (Jiang et al., 2021).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “The prompts can be combined using a uniform average [75, 166, 210] or a weighted average [75, 143, 166, 167].”
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, background): “We then benchmark these methods on two key tasks—confidence calibration and failure prediction—across five types of tasks (Commonsense, Arithmetic, Symbolic, Ethics and Professional Knowledge) and five widely-used LLMs, i.e., GPT-3 (Brown et al., 2020), GPT-3.”