Emergent Linear Representations in World Models of Self-Supervised Sequence Models

By Neel Nanda, Andrew Lee, Martin Wattenberg. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2023

353 citations recorded by the source corpus.

How do sequence models represent their decision-making process? Prior work suggests that Othello-playing neural network learned nonlinear models of the board state (Li et al., 2023a). In this work, we provide evidence of a closely related linear representation of the board. In particular, we show that probing for “my colour” vs. “opponent’s colour” may be a simple yet powerful way to interpret the model’s internal state. This precise understanding of the internal representations allows us to control the model’s behaviour with simple vector arithmetic. Linear representations enable significant interpretability progress, which we demonstrate with further exploration of how the world model is computed.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/bb26227a94ddb2b0088a23e2ec0a170c40bc4d78

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wes Gurnee
2. Max Tegmark
3. Victor Veitch
4. Kiho Park
5. Efstratios Gavves
6. Yo Joong Choe
7. Leonard Bereska

In Statistical and Nonlinear Physics

1. Wes Gurnee
2. Max Tegmark

Citations in the corpus

• Many-shot Jailbreaking (2024, neutral): “The effectiveness of the attack takes a hit on most domains, but improves steadily as a function of number of demonstrations.”
• Language Models Represent Space and Time (2023, background): “Subsequent work on Othello has shown these representations are also linear (Nanda et al., 2023).”
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024, neutral): “There are several lines of evidence that suggest that a significant fraction of the internal activations of language models are sparse, linear combination of vectors, each corresponding to meaningful features (Elhage et al., 2022; Gurnee et al., 2023; Olah et al., 2020; Park et al., 2023; Nanda et a”
• Mechanistic Interpretability for AI Safety - A Review (2024, background): “The interplay between observation and intervention facilitates a comprehensive understanding of neural network operations (Section 4.3).”
• Refusal in Language Models Is Mediated by a Single Direction (2024, neutral): “Various works show that directions in activation space have more “feature-like” properties than neurons do (Bolukbasi et al., 2016; Elhage et al., 2022; Geiger et al., 2024; Hernandez and Andreas, 2021; Li et al., 2021; Mikolov et al., 2013; Nanda et al., 2023; Park et al., 2023b).”
• The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “Measurement: (e.g., Nanda et al., 2023; Gurnee & Tegmark, 2023) Next is the idea that the probability of a concept value can be measured with a linear probe.”
• Understanding generative AI output with embedding models (2025, unknown): “Their internal representations have been shown to be sensitive to high-level concepts such as time and space ( 7 ), color ( 8 ), and more ( 9 , 10 ).”