Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

By Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt. International Conference on Learning Representations, 2022

1,133 citations recorded by the source corpus.

Research in mechanistic interpretability seeks to explain behaviors of machine learning models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models, or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task called indirect object identification (IOI). Our explanation encompasses 26 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches relying on causal interventions. To our knowledge, this investigation is the largest end-to-end attempt at reverse-engineering a natural behavior"in the wild"in a language model. We evaluate the reliability of our explanation using three quantitative criteria--faithfulness, completeness and minimality. Though these criteria support our explanation, they also point to remaining gaps in our understanding. Our work provides evidence that a mechanistic understanding of large ML models is feasible, opening opportunities to scale our understanding to both larger models and more complex tasks.

Subfields: Artificial Intelligence, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/6edd112383ad494f5f2eba72b6f4ffae122ce61f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

Citations in the corpus

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “…et al., 2021; Utama et al., 2020; Creager et al., 2021; Idrissi et al., 2022; Orgad & Belinkov, 2023), training classifiers with more favorable inductive biases (Kirichenko et al., 2023; Zhang et al., 2022; Iskander et al., 2024), or editing out undesired concepts (Iskander et al., 2023; Belrose…”
Can Transformers Learn to Solve Problems Recursively? (2023, methodology): “Mechanistic Interpretability Our analyses of the algorithms performed by our trained models was inspired by existing work in the relatively new field of mechanistic interpretability, and includes methods such as counterfactual patching[24], circuit tracing[39], automatic circuit discovery [7], and c”
Progress measures for grokking via mechanistic interpretability (2023, background): “If future mechanistic interpretability can only recover parts of the mechanism of larger models (as in Wang et al. (2022)) and can only generate comprehensive understanding of the mechanisms of smaller models, we might still be able to use our understanding from smaller models to guide the developme”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, methodology): “To do this, we start with some counterfactual corrupted data X̄i as in (Wang et al., 2022) and identify the minimum number of interventions needed to reproduce the behaviour of the model when run on the uncorrupted data.”
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (2024, background): “Moreover, it has been demonstrated that interpretable symbolic computations can be drawn from trained models (Clark et al., 2019; Tenney et al., 2019; Vig, 2019; Wang et al., 2022b).”
An Overview of Catastrophic AI Risks (2023, background): “Research can include analysis of small components [124, 125] of networks as well as investigation of how model internals produce a particular high-level behavior [126].”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “MI concerns itself with rigorously understanding the learned algorithms (circuits) utilized by neural networks [43, 44, 45, 46] in the hope of maintaining oversight and diagnosing failures of increasingly capable models [47].”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “On the other hand, mechanistic interpretability proposes editing models by intervening on individual neurons (Wang et al., 2022).”
Large Language Model Alignment: A Survey (2023, neutral): “, on a toy language model (typically one-to-four-layer Transformer language models), or with predefined simple tasks (Wang et al., 2022a; Elhage et al., 2021).”
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023, methodology): “, 2022) and interventions in the transformer computation (Wang et al., 2022; Haviv et al., 2023) to study the inner-workings of LMs.”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, neutral): “We used over 20% of the training compute of GPT-3 (Brown et al., 2020), saved about 20 Pebibytes (PiB) of activations to disk, and produced hundreds of billions of sparse autoencoder parameters in total.”