Rethinking eliminative connectionism.

By Gary F. Marcus. 1998

Humans routinely generalize universal relationships to unfamiliar instances. If we are told "if glork then frum," and "glork," we can infer "frum"; any name that serves as the subject of a sentence can appear as the object of a sentence. These universals are pervasive in language and reasoning. One account of how they are generalized holds that humans possess mechanisms that manipulate symbols and variables; an alternative account holds that symbol-manipulation can be eliminated from scientific theories in favor of descriptions couched in terms of networks of interconnected nodes. Can these "eliminative" connectionist models offer a genuine alternative? This article shows that eliminative connectionist models cannot account for how we extend universals to arbitrary items. The argument runs as follows. First, if these models, as currently conceived, were to extend universals to arbitrary instances, they would have to generalize outside the space of training examples. Next, it is shown that the class of eliminative connectionist models that is currently popular cannot learn to extend universals outside the training space. This limitation might be avoided through the use of an architecture that implements symbol manipulation.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “…trained on f(x)=x for only for even numbers, it will not extend the identity function to odd numbers, which lie outside the training distribution (Marcus, 1998) , To a human, it is obvious from a few examples that each output node, including the rightmost one, which represents the "1" bit…”
Environmental drivers of systematicity and generalization in a situated agent (2019, methodology): “Since the earliest days of research on neural networks, a recurring point of debate is whether neural networks exhibit generalisation beyond their training experience, in a systematic way (Smolensky, 1988; Fodor & Pylyshyn, 1988; Marcus, 1998; McClelland et al., 1987).”
Deep Learning: A Critical Appraisal (2018, background): “As discussed later in this article, generalization can be thought of as coming in two flavors, interpolation between known examples, and extrapolation, which requires going beyond a space of known training examples (Marcus, 1998a).”
Inductive biases for deep learning of higher-level cognition (2020, background): “…in small pieces which can be recomposed dynamically as needed (to reason, imagine or explain at an explicit level), one may achieve the kind of systematic generalization which humans enjoy and is obvious in natural language (Marcus, 1998, 2019; Lake and Baroni, 2017; Bahdanau et al., 2018).”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “It provides a natural way of achieving compositionality, which appears essential for systematic generalization, one of the areas where typical artificial neural networks (NNs) do not yet perform well (Fodor et al., 1988; Marcus, 1998; Lake & Baroni, 2018; Hupkes et al., 2020).”
A Usage-Based Approach to Recursion in Sentence Processing (2009, background): “Indeed, Hadley (1994) suggested that connectionist models could not afford the kind of generalization abilities necessary to account for human language processing (see Marcus, 1998, for a similar critique).”
Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks (2017, background): “…tested the ability of neural networks to solve tasks requiring compositional generalization, with mixed results (e.g., Christiansen & Chater, 1994; Marcus, 1998; Phillips, 1998; Chang, 2002; van der Velde et al., 2004; Botvinick & Plaut, 2006; Wong & Wang, 2007; Bow-ers et al., 2009; Botvinick &…”
Linguistic generalization and compositionality in modern artificial neural networks (2019, unknown): “, [19, 20, 21, 22, 23] among many others).”
Compositional Generalization for Primitive Substitutions (2019, background): “However, there have been critiques that neural networks do not have compositional generalization ability (Fodor and Pylyshyn, 1988; Marcus, 1998; Fodor and Lepore, 2002; Marcus, 2003; Calvo and Symons, 2014).”
Neuroscience-Inspired Artificial Intelligence. (2017, background): “These types of problems were previously argued to depend exclusively on symbol processing and variable binding and therefore beyond the purview of neural networks (Fodor and Pylyshyn, 1988; Marcus, 1998).”
Compositional generalization through meta sequence-to-sequence learning (2019, background): “Neural networks have long been criticized for lacking compositionality, leading critics to argue they are inappropriate for modeling language and thought [8, 24, 25].”
Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning (2001, background): “incapable of producing systematic behavior based on statistical learning of the environment (Marcus, 1998; Pinker & Prince, 1988).”
Learning Compositional Rules via Neural Program Synthesis (2020, background): “Fodor and Marcus have argued that systematic compositionality, while critical to human language and thought, is incompatible with classic neural networks (i.e., eliminative connectionism) (Fodor & Pylyshyn, 1988; Marcus, 1998; 2003).”
Human few-shot learning of compositional instructions (2019, background): “These advances have been largely driven by neural networks, a class of models that has been long criticized for lacking systematic compositionality (Fodor & Pylyshyn, 1988; Marcus, 1998; Fodor & Lepore, 2002; Marcus, 2003; Calvo & Symons, 2014).”