Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?

By William Merrill, Yoav Goldberg, Roy Schwartz, Noah A. Smith. 2021

83 citations recorded by the source corpus.

Abstract Language models trained on billions of tokens have recently led to unprecedented results on many NLP tasks. This success raises the question of whether, in principle, a system can ever “understand” raw text without access to some form of grounding. We formally investigate the abilities of ungrounded systems to acquire meaning. Our analysis focuses on the role of “assertions”: textual contexts that provide indirect clues about the underlying semantics. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation of languages that satisfy a strong notion of semantic transparency. However, for classes of languages where the same expression can take different values in different contexts, we show that emulation can become uncomputable. Finally, we discuss differences between our formal model and natural language, exploring how our results generalize to a modal setting and other semantic relations. Together, our results suggest that assertions in code or language do not provide sufficient signal to fully emulate semantic representations. We formalize ways in which ungrounded language models appear to be fundamentally limited in their ability to “understand”.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Prior works have suggested that pre-trained language models possess limited understanding of commonsense knowledge (Merrill et al., 2021; Talmor et al., 2021; Davis and Marcus, 2017) despite (1)We will share this following the anonymity period.”
A Review on Language Models as Knowledge Bases (2022, background): “However, despite succeeding in commonsense leaderboards, Merrill et al. (2021) suggest that those models fail to understand the underlying semantics leading them to commit trivial mistakes.”
Mapping Language Models to Grounded Conceptual Spaces (2022, background): “In principle, it is therefore possible for a text-only model’s conceptual space to be isomorphic to the “true” (i.e., grounded) conceptual space (Merrill et al., 2021).”
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color (2021, background): “The ﬁrst two evaluate the effect of a color term occurring more or less often as an adjectival modiﬁer, while the latter tests the hypothesis that assertions such as The banana is yellow could provide indirect grounding (Merrill et al., 2021), thereby leading to higher alignment.”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “In particular, Merrill et al. (2021) proves essential constraints on what can be learned from assertions alone, and more crucially, what cannot.”
When Combating Hype, Proceed with Caution (2021, background): “See Merrill et al. (2021) for some limits on how closely such a model can correspond to the real world and Bommasani et al. (2021, §2.6.3) for further discussion of the implications of Bender and Koller’s arguments for NLP.”
Synchromesh: Reliable code generation from pre-trained language models (2022, result): “This is in line with theoretical results in Merrill et al. (2021) showing that programming language semantics cannot be fully inferred from ungrounded data.”
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (2022, background): “The arguments range from philosophical (Bender & Koller, 2020) to mathematical (Merrill et al., 2021).”