Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

By Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, Martin Wattenberg. Neural Information Processing Systems, 2023

1,288 citations recorded by the source corpus.

We introduce Inference-Time Intervention (ITI), a technique designed to enhance the"truthfulness"of large language models (LLMs). ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads. This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark. On an instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from 32.5% to 65.1%. We identify a tradeoff between truthfulness and helpfulness and demonstrate how to balance it by tuning the intervention strength. ITI is minimally invasive and computationally inexpensive. Moreover, the technique is data efficient: while approaches like RLHF require extensive annotations, ITI locates truthful directions using only few hundred examples. Our findings suggest that LLMs may have an internal representation of the likelihood of something being true, even as they produce falsehoods on the surface.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/405f8f5f1c6df1b3343c812832479aad5180b65f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, methodology): “We use the model weights provided by Li et al. (2023a), denoted there as the synthetic model.”
Steering Language Models With Activation Engineering (2023, neutral): “Alongside contemporaneous work (Li et al. 2023b; Liu et al. 2023), our experiments suggest that activation engineering can flexibly retarget LLM behavior without damaging general performance.”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Recent empirical work on ELK includes a benchmark for measurement tampering (Roger et al., 2023), methods for discovering latent knowledge (Burns et al., 2023), and studies of honesty (Li et al., 2023; Pacchiardi et al., 2023).”
Large Language Models (2023, background): “” Is this somehow biasing the network towards certain texts and away from others (after all the internet corpus is hardly a reliable source of truth) ? Suppose we have a theory of how this works, how can we test it? Does the model “know” anything about the truth of statements? [25, 79] As has been m”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “Chuang et al. (2023) and Li et al. (2023c) propose methods that leverage the inconsistency of knowledge across different layers.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “HaluEval (Li et al., 2023a) hire human anno-tators to annotate the existence of hallucination in the model responses and list the corresponding spans.”
INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (2024, methodology): “It’s worth nothing that the ITI relies on training 1024 binary classifiers in TruthQA datasets, and they report the best performance (83.3) in the validation set.”
Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “It is well understood that adding feature vectors to the residual stream can modify behavior (Li et al., 2024a; Marks and Tegmark, 2023; Panickssery et al., 2023; Tigges et al., 2023; Turner et al., 2023; Zou et al., 2023a), although details on how and where to intervene vary (Jorgensen et al.,…”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, neutral): “However, unlike previous interpretability techniques like steering vectors (Turner et al., 2024; Li et al., 2023) or probing (Belinkov, 2022), sparse autoencoders can be highly expensive and difficult to train, limiting the ambition of interpretability research.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Figure 6 contrasts observed neural network models with hypothetical disentangled models, illustrating how a shared circuit pattern can emerge across different models trained on similar tasks and data, hinting at an underlying universality .”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Building on this idea, Li et al. (2023d) introduce Inference-Time Intervention (ITI).”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, methodology): “Recent studies (Meng et al., 2022; Dai et al., 2022; Li et al., 2023) highlight the role of middle and topmost layers in factual predictions and specific heads in truthfulness, respectively.”
A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models (2024, methodology): “Inference-Time Intervention (ITI): (Li et al., 2023a) introduce ITI, a technique designed to enhance the “truthfulness” of LLMs. ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads.”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, methodology): “Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022).”
Understanding the Effects of Iterative Prompting on Truthfulness (2024, methodology): “Another noteworthy approach involves the application of Inference-Time Intervention (ITI), which has shown considerable success in improving the truthfulness of model responses, marking a leap forward in model accuracy [18].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…the internal mechanisms of models can aid in directly editing model weights or intervening on internal activations in order to improve truthfulness [Li et al., 2023b], modify a model’s factual knowledge [Meng et al., 2023, 2022, Hernandez et al., 2023, Hase et al., 2023], or otherwise steer model…”