The Symbol Grounding Problem

By Stevan Harnad. 1990

There has been much discussion recently about the scope and limits of purely symbolic models of the mind and about the proper role of connectionism in cognitive modeling. This paper describes the "symbol grounding problem": How can the semantic interpretation of a formal symbol system be made intrinsic to the system, rather than just parasitic on the meanings in our heads? How can the meanings of the meaningless symbol tokens, manipulated solely on the basis of their (arbitrary) shapes, be grounded in anything but other meaningless symbols? The problem is analogous to trying to learn Chinese from a Chinese/Chinese dictionary alone. A candidate solution is sketched: Symbolic representations must be grounded bottom-up in nonsymbolic representations of two kinds: (1) "iconic representations" , which are analogs of the proximal sensory projections of distal objects and events, and (2) "categorical representations" , which are learned and innate feature-detectors that pick out the invariant features of object and event categories from their sensory projections. Elementary symbols are the names of these object and event categories, assigned on the basis of their (nonsymbolic) categorical representations. Higher-order (3) "symbolic representations" , grounded in these elementary symbols, consist of symbol strings describing category membership relations (e.g., "An X is a Y that is Z"). Connectionism is one natural candidate for the mechanism that learns the invariant features underlying categorical representations, thereby connecting names to the proximal projections of the distal objects they stand for. In this way connectionism can be seen as a complementary component in a hybrid nonsymbolic/symbolic model of the mind, rather than a rival to purely symbolic modeling. Such a hybrid model would not have an autonomous symbolic "module," however; the symbolic functions would emerge as an intrinsically "dedicated" symbol system as a consequence of the bottom-up grounding of categories' names in their sensory representations. Symbol manipulation would be governed not just by the arbitrary shapes of the symbol tokens, but by the nonarbitrary shapes of the icons and category invariants in which they are grounded.

Subfields: Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Philosophy

Citations in the corpus

Experience Grounds Language (2020, background): “Visual perception is a rich signal for modeling a vastness of experiences in the world that cannot be documented by text alone (Harnad, 1990).”
A simple neural network module for relational reasoning (2017, background): “Symbolic approaches to artificial intelligence are inherently relational [16, 5].”
Imitating Interactive Intelligence (2020, background): “Moreover, studying how to build agents that understand the “grounding” of language (Harnad, 1990) within their sensorimotor embodiment is both fundamentally interesting (Hill et al., 2019a; McClelland et al., 2019) and of obvious use for building robotic and other personal assistant artificial…”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “This symbol grounding [21] problem has largely focused on connecting language to static images.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “This in turn means that for a human or a machine to learn a language, they must solve what Harnad (1990) calls the symbol grounding problem .”
In Defense of Grid Features for Visual Question Answering (2020, background): “Note that end-to-end training with region features is challenging, since fine-tuning region locations likely requires additional grounding annotations [13] that are expensive and difficult to acquire.”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “An important limitation of symbolic AI relates to the so-called symbol grounding problem [14], and concerns the extent to which its representational elements are hand-crafted rather than learned from data (e.g. from sensory input).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “A major obstacle here is the symbol grounding problem [18, 19].”
On the Binding Problem in Artificial Neural Networks (2020, background): “Importantly, the distributed representations learned by neural networks are directly grounded in their input data, unlike symbols whose connection to real-world concepts is entirely subject to human interpretation (see symbol grounding problem; Harnad, 1990).”
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space (2019, background): “The latter approaches are less likely to generate crisp representations due to the dependence on images, but at the same time, they deal with two problems at the same time: representation learning and representation (symbol) grounding [20].”
Multimodal Distributional Semantics (2013, background): “…how models where the meaning of symbols (e.g., words) are entirely accounted for in terms of other symbols (e.g., other words) without links to the outside world (e.g., via perception) are deeply problematic , an issue that is often referred to as the symbol grounding problem (Harnad, 1990).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Apart from its practical applications, the grounding of language in perceptual data has long been a matter of scientific interest in ai (see Winograd, 1972; Harnad, 1990; Roy & Reiter, 2005, for a variety of theoretical views on the computational challenges of the perception-language interface).”
Towards a Visual Turing Challenge (2014, background): “Mos prominently, the latter is about finding (either with a hand-crafted set of rules or learnt-based approaches) a mapping between the linguistic fragments and the physical world [14, 15, 49], where as the question answering task is about an end-to-end system where we do not necessarily want to enf”
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language (2014, background): “Also, by connecting words and phrases to objects and events in the world, the semantics of language is grounded in perceptual experience (Harnad, 1990).”
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (2015, neutral): “These methods solve what Harnad [11] refers to as the symbol grounding problem, that of mapping linguistic elements to their corresponding manifestation in the external world.”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “One of the core challenges of Visual QA is the problem of grounding language : that is, associating the meaning of a language term with a speciﬁc perceptual input [36].”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, background): “However, compared to human semantic knowledge, these purely textual models, just like traditional symbolic AI systems (Harnad, 1990; Searle, 1984), are severely impoverished, suffering of lack of grounding in extra-linguistic modalities (Glenberg and Robertson, 2000).”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Learning to follow language instructions involves addressing a difﬁcult language grounding problem [15]: how do agents relate unstructured language l to their onboard perceptions s and actions a ?”
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World (2021, background): “…describing the initial world state, the action, and the action result. cent work arguing that no amount of language form could ever specify language meaning (McClelland et al., 2019; Bender and Koller, 2020; Bisk et al., 2020); connecting back to the Symbol Grounding Problem of Harnad (1990).”