The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

By Samuel Marks, Max Tegmark. arXiv.org, 2023

622 citations recorded by the source corpus.

Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

View Paper: https://www.semanticscholar.org/paper/d59523889679aee15992c4bf6e52b134186d07d3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Mengnan Du
2. Fan Yang
3. Ninghao Liu
4. Haiyan Zhao
5. Hanjie Chen
6. Andrew Lee
7. Chris Wendler
8. Huiqi Deng

Citations in the corpus

• Explainability for Large Language Models: A Survey (2023, background): “Then the overall uncertainty can be estimated based on the aggregation of token-level uncertainties.”
• Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “…(MI) aims to re-verse-engineer and thereby understand neural networks, using techniques such as circuit discovery (Nanda et al., 2023; Conmy et al., 2023), controlled task-specific training (Li et al., 2022; Marks and Tegmark, 2023), and causal tracing (Meng et al., 2022; Monea et al., 2023).”
• Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Burns et al. (2022); Marks and Tegmark (2023) find linear vectors that encode “truthfulness”.”
• Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “…representation of particular features such as harmlessness (Wolf et al., 2024; Zheng et al., 2024; Zou et al., 2023a), truth (Li et al., 2024a; Marks and Tegmark, 2023), humor (von Rütte et al., 2024), sentiment (Tigges et al., 2023), language (Bricken et al., 2023), topic (Turner et al.,…”