Language Models Represent Space and Time

By Wes Gurnee, Max Tegmark. International Conference on Learning Representations, 2023

357 citations recorded by the source corpus.

The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual"space neurons"and"time neurons"that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

Subfields: Artificial Intelligence, Cultural Studies

View Paper: https://www.semanticscholar.org/paper/740c783ac07039cf30b6d8a8f95e775b3297c79e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fan Yang
2. Ninghao Liu
3. Mengnan Du
4. A. Engel
5. Huiqi Deng
6. Tony Chiang
7. Shuaiqiang Wang
8. Haiyan Zhao

Citations in the corpus

• Mechanistic Interpretability for AI Safety - A Review (2024, background): “The interplay between observation and intervention facilitates a comprehensive understanding of neural network operations (Section 4.3).”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Recent methods [37, 44, 80, 85, 86] also induce language-based world models from large language models, but are restricted in textual space and struggle with grounding on physics [26, 59].”
• Explainability for Large Language Models: A Survey (2023, background): “The model also has specialized neurons that activate as a function of space or time, which aligns with the founding of factual knowledge in LLMs (Gurnee & Tegmark, 2023).”
• The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “Measurement: (e.g., Nanda et al., 2023; Gurnee & Tegmark, 2023) Next is the idea that the probability of a concept value can be measured with a linear probe.”
• RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue (2026, unknown): “Following standard layer-wise probing [22], we train a ridge regression at each layer and both frames to map the hidden state to the nomic-embed-text embedding [23] of PersonaPlex’s response, using single-turn general question answering 1 .”
• Understanding generative AI output with embedding models (2025, unknown): “Their internal representations have been shown to be sensitive to high-level concepts such as time and space ( 7 ), color ( 8 ), and more ( 9 , 10 ).”
• Understanding Generative AI Content with Embedding Models (2024, background): “Leveraging existing knowledge about appropriate datasets, these vector space representations have been found to implicitly understand concepts ranging from time and space [6] to color [7] and more [8, 9].”