Finding Neurons in a Haystack: Case Studies with Sparse Probing

By Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, Dimitris J. Bertsimas. Trans. Mach. Learn. Res., 2023

363 citations recorded by the source corpus.

Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.

View Paper: https://www.semanticscholar.org/paper/12910786da7a34c9ee26798fd81b0ed7b0e38789

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “For robustness, we also run all experiments on German, French, and Russian.”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, unknown): “…that suggest that a significant fraction of the internal activations of language models are sparse, linear combination of vectors, each corresponding to meaningful features (Elhage et al., 2022; Gurnee et al., 2023; Olah et al., 2020; Park et al., 2023; Nanda et al., 2023a; Mikolov et al., 2013).”
Language Models Represent Space and Time (2023, methodology): “For instance, Li et al. (2022) have shown that transformers trained with next token prediction to play the board game Othello learn explicit representations of the game state, with Nanda et al. (2023) subsequently showing these representations are linear.”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, background): “…neurons represent features (Wang et al., 2022; Sajjad et al., 2022; Bau et al., 2020), but features may more generally be represented by directions in a LLM’s latent space (i.e. linear combinations of neurons) (Dalvi et al., 2018; Gurnee et al., 2023; Cunningham et al., 2023; Elhage et al., 2022).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “These features are assumed to have the following properties: 1) Concept sparsity : Real-world concepts occur sparsely.”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Such framework has allowed researchers to better understand grokking (Nanda et al., 2023), superposition (Elhage et al., 2022b; Scherlis et al., 2022; Arora et al., 2018), or even individual neurons (Mu and Andreas, 2020; Antverg Belinkov, 2021; Gurnee et al., 2023).”