Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

By Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda B. Viégas, Hanspeter Pfister, Martin Wattenberg. International Conference on Learning Representations, 2022

503 citations recorded by the source corpus.

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create"latent saliency maps"that can help explain predictions in human terms.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/d5295f7ddcf281f3d30a7579d5ce482036a8e27c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Recent methods [37, 44, 80, 85, 86] also induce language-based world models from large language models, but are restricted in textual space and struggle with grounding on physics [26, 59].”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, methodology): “How do sequence models represent their decision-making process? Prior work suggests that Othello-playing neural network learned nonlinear models of the board state (Li et al., 2023a).”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, background): “One common approach is to train classifiers to extract specific concepts from hidden states, like part-of-speech and syntactic structure (Hewitt and Manning, 2019; Tucker et al., 2021; Li et al., 2022).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “The interplay between observation and intervention facilitates a comprehensive understanding of neural network operations (Section 4.3).”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “(Li et al., 2022) trained a model to predict the next move in the board game Othello, and were able to probe for an emergent representation of the board state.”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, background): “Experiments in [26] have shown that training sequence models on a specific task gives rise to an internal world representation within the model.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “More similar to our study, Li et al. (2023b) showed that while the LMs they investigated seem to be able to perform some reasoning in counterfactual worlds, this is largely affected by superficial lexical cues.”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “…for interpreting (probing) (e.g., Alain & Ben-gio, 2017; Kim et al., 2018; nostalgebraist, 2020; Rogers et al., 2021; Belinkov, 2022; Li et al., 2022; Geva et al., 2022; Nanda et al., 2023) and controlling (steering) (e.g., Wang et al., 2023; Turner et al., 2023; Merullo et al.,…”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “…(MI) aims to re-verse-engineer and thereby understand neural networks, using techniques such as circuit discovery (Nanda et al., 2023; Conmy et al., 2023), controlled task-specific training (Li et al., 2022; Marks and Tegmark, 2023), and causal tracing (Meng et al., 2022; Monea et al., 2023).”
Large Language Models (2023, methodology): “My own opinion is that there is no barrier in principle to LLMs constructing internal non-linguistic models of the world, and the work [78] on Othello-GPT discussed in §7 is a nice demonstration of what is possible.”
Language Models Represent Space and Time (2023, background): “…that such models are learnable from next-token prediction comes from GPT-style models trained on chess (Toshniwal et al., 2022) and Othello games (Li et al., 2022) which were shown to have explicit representations of the board and game state, with further work showing these representations are…”
Steering Language Models With Activation Engineering (2023, unknown): “Finally we plot these for both modified and unmodified models over a range of K values.”
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023, methodology): “This ﬁner-grained intervention contrasts with previous transformer activation editing meth-ods (Hernandez et al., 2023; Li et al., 2023) that work on the residual stream after the MLP. Working on attention heads’ activation spaces enables us to leave irrelevant heads out to be less intrusive to…”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, methodology): “Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022).”
Understanding Generative AI Content with Embedding Models (2024, background): “Leveraging existing knowledge about appropriate datasets, these vector space representations have been found to implicitly understand concepts ranging from time and space [6] to color [7] and more [8, 9].”