Implicit Representations of Meaning in Neural Language Models

By Belinda Z. Li, Maxwell Nye, Jacob Andreas. 2021

240 citations recorded by the source corpus.

Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 transformer language models, we identify contextual word representations that function as *models of entities and situations* as they evolve throughout a discourse. These neural representations have functional similarities to linguistic models of dynamic semantics: they support a linear readout of each entity’s current properties and relations, and can be manipulated with predictable effects on language generation. Our results indicate that prediction in pretrained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

Citations in the corpus

Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, neutral): “In response, Li et al. (2021) provide evidence that pre-trained language models do indeed construct approximate representations of the semantics of the situations they describe in text.”
Program Synthesis with Large Language Models (2021, background): “On the other hand, recent work has provided evidence that, in some natural language contexts, pre-trained Transformers are able to implicitly construct approximate representations of the semantics of the worlds they describe in text [Li et al., 2021].”
Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “Finally, they appear to be capable of basic simulation, modeling changes in entity states and relations described by text (Li et al., 2021).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “…Radford et al., 2019; Raffel et al., 2019; Brown et al., 2020) but the learned representations can also be used to model relations of entities (Li et al., 2021), retrieve matching visual features (Ilharco et al., 2020), synthesize code from docstrings (Hendrycks et al., 2021; Chen et al.,…”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Additionally, we encourage further investigations into probing the latent knowledge within LLMs (Tenney et al., 2019; Li et al., 2021) or examining how LLMs handle entities and people (Onoe et al.”
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (2022, background): “Recent evidence suggests language models can develop internal representations for very simple concepts, such as color, direction Abdou et al. (2021); Patel & Pavlick (2022), or tracking boolean states during synthetic tasks (Li et al., 2021) (see Related Work (section 6) for more detail).”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, background): “Recent work (Li et al., 2021) has shown that large pre-trained neural models learn to approximately represent certain types of structured semantic information.”
Towards Reasoning in Large Language Models: A Survey (2022, methodology): “In fact, there is also an ongoing debate about whether language models can actually understand language or capture meaning (Bender and Koller, 2020; Li et al., 2021; Manning, 2022; Piantasodi and Hill, 2022).”
What Can a Generative Language Model Answer About a Passage? (2021, background): “However, there is some evidence that deep neural language models may rediscover a similar NLP pipeline (Tenney et al., 2019; Li et al., 2021).”
Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “Various works show that directions in activation space have more “feature-like” properties than neurons do (Bolukbasi et al., 2016; Elhage et al., 2022; Geiger et al., 2024; Hernandez and Andreas, 2021; Li et al., 2021; Mikolov et al., 2013; Nanda et al., 2023; Park et al., 2023b).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “The ability to internally represent the board state in games like Othello (Li et al., 2023a; Nanda et al., 2023b), create linear abstractions of spatial and temporal data (Gurnee & Tegmark, 2024), and structure complex representations of mazes, demonstrating an understanding of maze topology and pat”
The rise and potential of large language model based agents: a survey (2025, unknown): “Although intuitively the paradigm of next token prediction in LLMs may not possess intention or desire, research has shown that they can implicitly generate representations of these states and guide the model’s inference process [105–107].”
Language Models Represent Space and Time (2023, background): “Probing To find evidence of spatial and temporal representations in LLMs, we use the standard technique of probing Alain & Bengio (2016); Belinkov (2022), which fits a simple model on the network activations to predict some target label associated with labeled input data.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “Much prior work has investigated the extent to which LMs acquire a grounded understanding of the world through text-only training (Piantadosi and Hill, 2022; Zhang et al., 2020; Ilharco et al., 2021; Li et al., 2021; i.a. ).”
Talking about Large Language Models (2022, background): “In chain-of-thought prompting, for example, a prompt preﬁx is submitted to the model, before the user’s query, containing a few examples of multi-step reasoning, with all the intermediate steps explicitly spelled out (Nye et al., 2021; Wei et al., 2022b).”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, background): “Substantial previous work has studied whether LLMs encode world models in their representations (Li et al., 2023a; 2021; Abdou et al., 2021; Patel & Pavlick, 2022).”
The debate over understanding in AI’s large language models (2022, background): “to develop new kinds of benchmarks and probing methods that can yield insight into the mechanisms of diverse types of intelligence and understanding, including the novel forms of “exotic, mind-like entities” (79) we have created, perhaps along the lines of some promising initial efforts (80, 81).”
Natural Language to Code Translation with Execution (2022, result): “On the other hand, our results are relevant to the recent discussion between meaning and form in pretrained language models (Bender and Koller, 2020; Li et al., 2021).”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Li et al. (2021) train sequence models on a synthetic task, and uncover world models in their activations.”