SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

By Potsawee Manakul, Adian Liusie, Mark John Francis Gales. Conference on Empirical Methods in Natural Language Processing, 2023

1,075 citations recorded by the source corpus.

Generative Large Language Models (LLMs) such as GPT-3 are capable of generating highly fluent responses to a wide variety of user prompts. However, LLMs are known to hallucinate facts and make non-factual statements which can undermine trust in their output. Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules. In this work, we propose"SelfCheckGPT", a simple sampling-based approach that can be used to fact-check the responses of black-box models in a zero-resource fashion, i.e. without an external database. SelfCheckGPT leverages the simple idea that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset, and manually annotate the factuality of the generated passages. We demonstrate that SelfCheckGPT can: i) detect non-factual and factual sentences; and ii) rank passages in terms of factuality. We compare our approach to several baselines and show that our approach has considerably higher AUC-PR scores in sentence-level hallucination detection and higher correlation scores in passage-level factuality assessment compared to grey-box methods.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7c1707db9aafd209aa93db3251e7ebd593d55876

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, methodology): “Comparison of fabrication rates, e.g., using approaches like SelfCheckGPT (Manakul et al., 2023), would also improve on the current analysis.”
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, neutral): “We examine how well model editors can mitigate hallucination in autoregressive language models using a new dataset released with SelfCheckGPT [27].”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “Researches on mitigating current LLMs’ hallucination issues often focuses on several aspects, including refining the training process, us-ing larger and more diverse datasets [24], or implementing post-training evaluation [11] and correction mechanisms [33, 34].”
Large Language Models: A Survey (2024, neutral): “3) Self-Consistency: Self-Consistency [159] utilizes an ensemble-based method, where the LLM is prompted to generate multiple responses to the same query.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “Norm Entropy (Malinin & Gales, 2021) leverages geometric mean token probability, where we calculate confidence as the arithmetic mean negative log-probability, given by ConfNorm Entropy = − 1N ∑N i log p(yi|y i).”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, background): “Recent strategies involve inference-time self-consistency checks (Manakul et al., 2023), multi-agent debating (Du et al., 2023; Liang et al., 2023), and inference-time intervention using human labels (Li et al., 2023).”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, neutral): “Accordingly, techniques of letting these legal AI tools “check themselves”—which have become popular in other AI evaluation pipelines (Manakul et al., 2023; Mündler et al., 2023; Zheng et al., 2023)—are not suitable for this application.”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “Although LLMs are used in various fields, a significant challenge is the phenomenon of ’ hallucination ’, where language models generate outputs that are plausible-sounding but factually incorrect or not referable in the input data [99], [100].”
Prompt Injection attack against LLM-integrated Applications (2023, background): “Since LLMs are trained on vast crawled datasets, they have been shown to carry potential risks of generating contentious or biased content, or even perpetuating hate speech and stereotypes [8,17,34,35,62].”
A Survey of Hallucination in Large Foundation Models (2023, methodology): “SELFCHECKGPT (Manakul et al., 2023), is a method for zero-resource black-box hallucination detection in generative LLMs.”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, neutral): “SelfCheckGPT (Manakul et al., 2023) addresses this problem by instead sampling multiple answers.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “We also compare with Self-check LM , a method from a concurrent work by Manakul et al. (2023).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “Uncertainty serves as a valuable indicator for detecting and mitigating hallucinations during the inference process (Manakul et al., 2023).”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, methodology): “Prior methods proposed to verify the factuality of statements include zero-resource approaches (Manakul et al., 2023; Kadavath et al., 2022; Agrawal et al., 2023; Azaria and Mitchell, 2023; Min et al., 2023), which operate mostly in a black-box fashion, and resource-enriched approaches (Thorne et al”
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023, methodology): “At the same time, LLMs also tend to generate responses that are factually incorrect or completely fabricated ( hallucinations ) (Manakul et al., 2023; Peng et al., 2023; Azaria and Mitchell, 2023).”
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023, background): “Ac-a The best answer is Answer 1. cording to recent work (Manakul et al., 2023), hallucinations are likely to appear in diverged and conflicting responses of LLMs. Based on the filtered user queries and ChatGPT responses, we invite human labelers to annotate whether the response contains…”