Inductive biases for deep learning of higher-level cognition

By Anirudh Goyal, Yoshua Bengio. 2020

474 citations recorded by the source corpus.

A fascinating hypothesis is that human and animal intelligence could be explained by a few principles (rather than an encyclopaedic list of heuristics). If that hypothesis was correct, we could more easily both understand our own intelligence and build intelligent machines. Just like in physics, the principles themselves would not be sufficient to predict the behaviour of complex systems like brains, and substantial computation might be needed to simulate human-like intelligence. This hypothesis would suggest that studying the kind of inductive biases that humans and animals exploit could help both clarify these principles and provide inspiration for AI research and neuroscience theories. Deep learning already exploits several key inductive biases, and this work considers a larger list, focusing on those which concern mostly higher-level and sequential conscious processing. The objective of clarifying these particular principles is that they could potentially help us build AI systems benefiting from humans’ abilities in terms of flexible out-of-distribution and systematic generalization, which is currently an area where a large gap exists between state-of-the-art machine learning and human intelligence.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Model Architectures

Citations in the corpus

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “It has been proposed that causal reasoning will be central to move modern machine learning algorithms beyond their current shortcomings, such as their lack of robustness, transferability and interpretability (Pearl, 2019; Schölkopf, 2019; Schölkopf et al., 2021; Goyal and Bengio, 2021).”
Pretrained Transformers as Universal Computation Engines (2021, background): “Language might also be a natural choice of model for this blackboard, as there are hypotheses that language may serve as a good multipurpose high-level representation for cognitive behavior and conscious planning (Andreas, Klein, and Levine 2017; Goyal and Bengio 2020).”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, background): “CL in the brain is governed by a rich set of neurophysiological processes that harbor different types of knowledge, and conscious processing integrates them coherently (Goyal & Bengio, 2020).”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, background): “slow learning of stationary aspects between tasks and fast learning of task-specific aspects (Goyal and Bengio, 2020).”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, methodology): “We compare the proposed approach against SCOFF (Goyal et al., 2020).”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, methodology): “…with a homogeneous architecture as in standard neural networks, we adopt recently proposed approaches (Goyal et al., 2019; Mittal et al., 2020; Goyal et al., 2020; Rahaman Input images processed through an encoder, and the embedded mission instruction are passed through a set of recurrent…”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Although the self-attention [21] in Transformers is effective in computing a representation of the sequence, an enormous amount of training data is still needed since Transformers often lack induction biases [18], [22], [23].”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “As a result, there has been a recent surge of interest in learning causal models for deep reinforcement learning [de Haan et al., 2019, Dasgupta et al., 2019, Nair et al., 2019, Goyal et al., 2019, Goyal and Bengio, 2020, Rezende et al., 2020, Wang et al., 2021, Schölkopf et al., 2021].”
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts (2021, background): “While models are desired to generalize to arbitrary OOD scenarios, the no-free-lunch theorem for machine learning [3, 12, 35] demonstrates that there is no entirely generalpurpose learning algorithm, and that any learning algorithm implicitly or explicitly will generalize better on some distribution”
Neural Production Systems (2021, methodology): “Here we have focused on perceptual inference problems and proposed NPS, a neural instantiation of production systems by introducing an important inductive bias in the architecture following the proposals of Bengio (2017); Goyal & Bengio (2020); Ke et al. (2021).”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “It is also a burgeoning topic in machine learning and artiicial intelligence [6, 16, 65, 75, 144, 214, 255, 265], where it has been argued that a consideration for causality is crucial for reasoning about the world.”
Large Language Models (2023, background): “Still both similarities and differences are interesting; can we make the analogies with cognitive psychology more precise? One analogy [17, 50], is with the well known concept of “fast and slow thinking” in behavioral psychology [66].”
On the link between conscious function and general intelligence in humans and machines (2022, methodology): “Goyal & Bengio (2020) extends this interpretation to incorporate aspects of embodied intelligence, with a focus on the role of causality within the factor graphs.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “It has been argued that integrating System 2 processing could similarly improve AI systems (Goyal & Bengio, 2020; Garcez & Lamb, 2020), and here we explore this idea as applied to neural sequence models.”