Teaching Models to Express Their Uncertainty in Words

By Stephanie C. Lin, Jacob Hilton, Owain Evans. 2022

778 citations recorded by the source corpus.

We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g."90% confidence"or"high confidence"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own answers in natural language. For testing calibration, we introduce the CalibratedMath suite of tasks. We compare the calibration of uncertainty expressed in words ("verbalized probability") to uncertainty extracted from model logits. Both kinds of uncertainty are capable of generalizing calibration under distribution shift. We also provide evidence that GPT-3's ability to generalize calibration depends on pre-trained latent representations that correlate with epistemic uncertainty over its answers.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, methodology): “In fact, some recent work [40, 39] have explored model fine-tuning and exhibiting the effectiveness of Binary for uncertainty calibration.”
Large language models encode clinical knowledge (2022, neutral): “While uncertainty measures over LLM output sequences remains an open area of research [36, 51], here we explored a simple proxy as an initial approach to measuring the relationship between LLM uncertainty and statement accuracy.”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, background): “…like classification or regression (Lakshminarayanan et al., 2017; Gal & Ghahramani, 2016; Hern´andez-Lobato & Adams, 2015; Abdar et al., 2021a), it bears specific challenges for NLG and has attracted limited attention until recently (Lin et al., 2022a; Kuhn et al., 2023; Malinin & Gales, 2021).”
Detecting hallucinations in large language models using semantic entropy (2024, neutral): “…in which a similar ‘symptom’ is caused by the following different mechanisms: when LLMs are consistently wrong as a result of being trained on erroneous data such as common misconceptions 13 ; when the LLM ‘lies’ in pursuit of a reward 14 ; or systematic failures of reasoning or generalization.”
Measuring short-form factuality in large language models (2024, background): “Other work has attempted to improve the calibration of language models (Lin et al., 2022b; Agrawal et al., 2024), which could lead to increased adoption of language models in real-world settings.”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “This is one area where supervised language-model-based uncertainty tools (Lin et al., 2022a; Kadavath et al., 2022) might enable future improvements that handle this challenge better.”
Can Foundation Models Wrangle Your Data? (2022, background): “Prior work demonstrates that a FM can “learn to express uncertainty about its own answers in natural language” [61].”
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023, background): “Several studies have examined the calibration of large LMs (Lin et al., 2022a; Park and Caragea, 2022; Kadavath et al., 2022; Xiao et al., 2022; Kuhn et al., 2023), finding that combining large pre-trained LMs with temperature scaling (Guo et al., 2017) produces very well-2 calibrated predictions…”
Aligning Large Multimodal Models with Factually Augmented RLHF (2023, background): “With our current template design, we aim to prompt crowdworkers to identify potential hallucinations in the model’s responses.”
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, background): “Principally, there are two indicators available in LLMs for gauging a model’s uncertainty: the textual output generated by the model (Lin et al., 2022) and the consistency among its multiple responses to the same question (Gawlikowski et al.”
Prompting GPT-3 To Be Reliable (2022, positive): “In the context of prompting LLMs, several previous works have explored their reliability.”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “They are widely known to “hallucinate” information (Ji et al., 2022; S. Lin et al., 2022b; Maynez et al., 2020), producing outputs that are plausible—even convincing— but incorrect.”
The alignment problem from a deep learning perspective (2022, background): “For example: • When answering questions, an RLHF-trained policy would beneﬁt from “knowing what it knows” in order to provide well-calibrated answers [Kadavath et al., 2022, Lin et al., 2022].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “In this work, we address these issues and improve the reliability of expressed uncertainty [122, 75, 132] by interacting with external tools like search engines, see §4.”
LLM-confidence reranker: A training-free approach for enhancing retrieval-augmented generation systems (2026, unknown): “For example, Lin et al. (2022) enabled models to express uncertainty in natural language via fine-tuning.”