The Internal State of an LLM Knows When its Lying

By Amos Azaria, Tom Michael Mitchell. Conference on Empirical Methods in Natural Language Processing, 2023

728 citations recorded by the source corpus.

While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/f406aceba4f29cc7cfbe7edb2f52f01374486589

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Several studies have demonstrated that LLM’s activations encapsulate an internal belief related to the truthfulness of its generated statements (Burns et al., 2022; Azaria and Mitchell, 2023).”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, background): “Recently, Azaria and Mitchell (2023) trained a multi-layer perception classifier where an LLM’s hidden representations are used as inputs to predict the truthfulness of a sentence.”
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023, background): “At the same time, LLMs also tend to generate responses that are factually incorrect or completely fabricated ( hallucinations ) (Manakul et al., 2023; Peng et al., 2023; Azaria and Mitchell, 2023).”
A Survey of Hallucination in Large Foundation Models (2023, methodology): “The Internal State of an LLM Knows When its Lying (Azaria and Mitchell, 2023) Classification Manual Accuracy”
Understanding the Effects of Iterative Prompting on Truthfulness (2024, background): “For instance, studies have demonstrated the potential of analyzing an LLM’s internal state to accurately gauge the truthfulness of its outputs, highlighting a promising direction for enhancing reliability [17].”
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (2023, methodology): “For example, in this paper we only consider censorship based on inputs and outputs of a model and do not consider methods that rely on internal representations of LLMs [4, 7, 27].”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, background): “Recent analyses, however, suggest that existing models struggle with detecting hallucination when using standard prompting strategies (Li et al., 2023; Azaria and Mitchell, 2023).”
INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (2024, methodology): “Therefore, hallucination detection, that is, accurately detecting and rejecting responses when hallucinations occur in LLMs, has attracted more and more attention from the academic community (Azaria & Mitchell, 2023; Ren et al., 2022; Kuhn et al., 2022).”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “…to verify the factuality of statements include zero-resource approaches (Man-akul et al., 2023; Kadavath et al., 2022; Agrawal et al., 2023; Azaria and Mitchell, 2023; Min et al., 2023), which operate mostly in a black-box fashion, and resource-enriched approaches (Thorne et al., 2018; Guo…”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, methodology): “For the curated datasets 3 we observe clear linear 2 Burns et al. (2023); Azaria & Mitchell (2023); Levinstein & Herrmann (2023) do test the transfer classification accuracy of probes (with mixed results), but do not perform any causal mediation experiments.”
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, background): “In different contexts, the definition and criteria for hallucination vary, and the annotation of hallucina-310 tion is not always straightforward.”