On the Binding Problem in Artificial Neural Networks

By Klaus Greff, Sjoerd van Steenkiste, J. Urgen Schmidhuber. 2020

324 citations recorded by the source corpus.

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around forming meaningful entities from unstructured sensory inputs (segregation), maintaining this separation of information at a representational level (representation), and using these entities to construct new inferences, predictions, and behaviors (composition). Our analysis draws inspiration from a wealth of research in neuroscience and cognitive psychology, and surveys relevant mechanisms from the machine learning literature, to help identify a combination of inductive biases that allow symbolic information processing to emerge naturally in neural networks. We believe that a compositional approach to AI, in terms of grounded symbol-like representations, is of fundamental importance for realizing human-level generalization, and we hope that this paper may contribute towards that goal as a reference and inspiration.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Model Architectures

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “Prompts like the latter require the model to perform variable binding (Smolensky, 1990; Greff et al., 2020) – it is the hedgehog that is in the christmas sweater, not the dog.”
Toward Causal Representation Learning (2021, background): “Note that the problem of object-centric representation learning [11], [40], [84], [87], [88], [138], [155], [160], [255], [262] can also be considered a special case of disentangled factorization as discussed here.”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, background): “they are self-contained and reusable independent of context) for solving a particular task, in the sense that they can be separately intervened upon or reasoned with (Greff et al., 2020).”
Conditional Object-Centric Learning from Video (2021, background): “Object-centric representations have the potential to greatly improve sample efficiency, robustness, generalization to new tasks, and interpretability of machine learning algorithms (Greff et al., 2020).”
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022, background): “Mask-Rich especially probes models’ ability to handle complex attribute binding and inclusion [12].”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “The increased popularity of structure discovery in deep learning is not without sound motivation, with arguments that disentangled, structured, and symbolic representations are key to the next generation of AI, as well as robust cross-domain performance, transfer learning, and interpretability [17, ”
Linear Transformers Are Secretly Fast Weight Programmers (2021, methodology): “This concept was called synaptic modulation (von der Malsburg, 1981), a method for variable binding in neural networks (see e.g. the recent survey by Greff et al. (2020)), or dynamic connections (Feldman, 1982).”