Toy Models of Superposition

By Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, T. Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen and 6 more. arXiv.org, 2022

947 citations recorded by the source corpus.

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in"superposition."We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Safety, Robustness & Interpretability

View Paper: https://arxiv.org/abs/2209.10652

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Safety, Robustness & Interpretability

Citations in the corpus

Refusal in Language Models Is Mediated by a Single Direction (2024, positive): “It is widely hypothesized that LLMs represent features, or concepts, as linear directions in activation space (Bolukbasi et al., 2016; Elhage et al., 2022; Mikolov et al., 2013; Park et al., 2023b).”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “Training dynamics in residual models can be explained by shallow paths through the computational graph (Veit, Wilber, and Belongie, 2016).”
Large Language Model Alignment: A Survey (2023, background): “Feature superposition in neural networks explains the phenomenon of neuron polysemanticity where a neuron corresponds to several unrelated features (Elhage et al., 2022b).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “Recent work in toy models demonstrate that this tension manifests in a spectrum of representations: dedicated dimensions or neurons for the most prevalent and important features with increasing levels of superposition—and hence decreasing levels of sparsity—for the long tail of rarer or less importa”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “…relates to many other fields in AI research, including compressed sensing (Elhage et al., 2022b), modularity, adversarial robustness, continual learning, network compression (Räuker et al., 2023), neurosymbolic reasoning, trojan detection, and program synthesis (Casper, 2023; Michaud et al., 2024).”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “A fundamental challenge in NN inter-pretability is that individual neurons are rarely interpretable (Elhage et al., 2022).”
Scaling and evaluating sparse autoencoders (2024, positive): “E.7 Clustering latents [Elhage et al., 2022] discuss how underlying features may lie in distinct sub-spaces.”
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, methodology): “In Appendix D, we present additional evaluation results covering • Sparsity-Fidelity trade-off for more SAEs • Studying SAE performance as a function of token position.”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “Elhage et al. (2022b) investigate why polysemanticity might arise and hypothesise that it may result from models learning more distinct features than there are dimensions in the layer.”
Language Models Represent Space and Time (2023, background): “…many results and ideas from the inter-pretability literature (R ¨ auker et al., 2023), especially in topics related to probing (Belinkov, 2022), BERTology (Rogers et al., 2021), the linearity hypothesis and superposition (Elhage et al., 2022b), and mechanistic interpretability (Olah et al., 2020).”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, background): “Intuitively, LR treats the small projection of θ f onto θ t as significant, and adjusts the probe direction to have less “interference” (Elhage et al., 2022) from θ f .”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “In the context of language models, the “Linear Representation Hypothesis” is the idea that high-level concepts are represented linearly in the representation space of a model (e.g., Mikolov et al., 2013c; Arora et al., 2016; Elhage et al., 2022).”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “However, this only elicits a single important direction, (4)For simplicity we assume the “features as directions” hypothesis (Elhage et al., 2022), which defines a “feature” to be the one-dimensional subspace spanned by a unit vector v.”
Sparse Prototype Code Underlies Classification and Prediction Across Modalities (2026, unknown): “A growing body of work in mechanistic interpretability addresses this challenge by extracting sparse, human-interpretable features from network activations [Elhage et al., 2022, Bricken et al., 2023, Huben et al., 2024, Templeton et al., 2026].”