Language Models (Mostly) Know What They Know

By Saurav Kadavath, Tom Conerly, Amanda Askell, T. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Z. Dodds, Nova Dassarma, Eli Tran-Johnson and 27 more. 2022

1,782 citations recorded by the source corpus.

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability"P(True)"that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict"P(IK)", the probability that"I know"the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “We also leverage the fact that language models can make well-calibrated choices [Kadavath et al., 2022] to turn AI choices into calibrated preference labels.”
Emergent Abilities of Large Language Models (2022, neutral): “Figure 2A shows an arithmetic benchmark that tests 3-digit addition and subtraction, as well as 2-digit multiplication.”
Measuring Progress on Scalable Oversight for Large Language Models (2022, positive): “These results are consistent with Kadavath et al. (2022)’s report that our plain pretrained language models tend to be very well calibrated on multiple-choice question answering, but that their calibration degrades after RLHF training for helpfulness, which we use for all runs in this paper.”
Large Language Models Can Self-Improve (2022, neutral): “Moreover, a recent study (Kadavath et al., 2022) shows that language model calibration increases with model size.”
Faithful Reasoning Using Large Language Models (2022, background): “Kadavath et al. (2022) also investigates when LMs “know” the answer.”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “For larger models, these uncertainty estimates happen to be well-calibrated and allow for safer deployment in high-stakes scenarios (Kadavath et al., 2022).”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, neutral): “For example, KNN-KMs [56] first retrieves 𝑘 most relevant contexts from the datastore based on the given query, and computes a neighbor distribution based on the distance.”
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023, background): “Kadavath et al. (2022) ﬁnd language models can generate and then self-evaluate their own answers with high accuracy.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Understanding when we should not trust LMs’ outputs is also crucial to safely deploying them in real-world applications (Kadavath et al., 2022).”
Large language models encode clinical knowledge (2022, neutral): “While uncertainty measures over LLM output sequences remains an open area of research [36, 51], here we explored a simple proxy as an initial approach to measuring the relationship between LLM uncertainty and statement accuracy.”
Defending ChatGPT against jailbreak attack via self-reminders (2023, neutral): “32 investigates the ability of LLMs to evaluate the validity of their response and predict their ability to answer questions, and ref.”
The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, positive): “At the early stage, the policy demonstrates high covariance on training data, implicating the policy’s confidence is well-calibrated (Kadavath et al., 2022), thus can safely exploit trajectories with high confidence, strengthening belief and minimize entropy (Zuo et al., 2025; Zhang et al., 2025;…”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, neutral): “questions than pre-trained models do.[44]”
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, methodology): “We follow the prompts provided in [23, 20].”