Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

By Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, Bryan Hooi. International Conference on Learning Representations, 2023

1,033 citations recorded by the source corpus.

Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.

Subfields: Accounting, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/8f7297454d7f44365b9bcda5ebb9439a43daf5e6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Political Science and International Relations

1. Dawei Yin

Citations in the corpus

• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, neutral): “In this part, we will briefly review studies with these two types of generators: parameter-accessible (white-box) and parameter-inaccessible (black-box).”
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Miao et al. (2023) introduced a sentence-level hal-lucination detection dataset by generating synthetic Wikipedia articles using GPT-3, based on concepts from the WikiBio dataset.”
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Jin et al. [121] conducted a comprehensive study on LLMs’ understanding of binary code semantics [122] across different architectures and optimization levels, providing key insights for future research in this area.”
• RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, background): “Please refer to Appendix C for more statistical in-316 formation about these annotations.”
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “Generally speaking, methods for estimating the uncertainty of LLMs can be categorized into three types (Xiong et al., 2023), as listed below.”
• A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “With some uncertainty estimation techniques [112, 191, 198], this can ensure that even in dynamic or edge case scenarios, the system can make sound judgments while keeping the user informed, therefore building confidence in autonomous technology.”
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “Other non-math, symbolic or logical datasets that benefit from CoT are Commitment Bank (de Marneffe et al., 2019) and the task of eliciting verbalized confidence (Xiong et al., 2024).”
• Explainability for Large Language Models: A Survey (2023, methodology): “Extensive experiments on multiple medical image datasets demonstrate the effectiveness and explainability of this training-free diagnostic pipeline.”
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023, background): “For instance, Kamath et al. (2020) examines the selective question-answering task as a multiple-choice problem, reducing it to a de facto classification task rather than directly engaging with free-form generation.”
• ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, methodology): “A large body of recent work has focused on improving the reasoning capabilities of Large Language Models (LLMs) by imitating various human cognitive processes (Wang and Zhao, 2023; Park et al., 2023; Sumers et al., 2023; Ye et al., 2023).”
• BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals (2026, unknown): “Xiong et al. [20] benchmark such elicitation across five models and five kinds of task and find the verbalised numbers overconfident, clustered high, and only nar-rowly behind methods that read the model’s own logits.”
• LLM-confidence reranker: A training-free approach for enhancing retrieval-augmented generation systems (2026, unknown): “Meanwhile, Xiong et al. (2024) further systematically proposed a general black-box confidence elicitation framework, combining human-inspired prompting strategies, diverse sampling methods, and confidence aggregation mechanisms, which can significantly improve LLMs’ performance in tasks like…”