Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

By Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning. Conference on Empirical Methods in Natural Language Processing, 2023

810 citations recorded by the source corpus.

A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ab4ce5dda7ad4d9032995c9c049a89d65723c6aa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, methodology): “Directly using confidence scores as the voting weights is less effective due to the overconfidence problem of LLMs (Xiong et al., 2023b; Tian et al., 2023; Mielke et al., 2022).”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “Challenges remain and may, in some cases, be ampliﬁed when multi-agent systems become available (Figure 6).”
A Survey on Evaluation of Large Language Models (2023, background): “By summarizing and analyzing existing efforts on LLMs evaluation, we may identify the current success and failure cases of LLMs, derive new trends for evaluation protocols, and most importantly, propose new challenges and opportunities for future research.”
Understanding the Effects of Iterative Prompting on Truthfulness (2024, background): “This could be due to findings suggesting that RLHF leads to poor calibration and sycophantic behavior [47].”