Measuring abstract reasoning in neural networks

By Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, Timothy P. Lillicrap. 2018

408 citations recorded by the source corpus.

Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.

Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “It was subsequently shown that an architecture based on relation networks could be successfully applied to certain analogical reasoning tasks [39].”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “We train the same relational model (Santoro et al., 2018) as in (van Steenkiste et al., 2019) (with identical hyperparameters) on the frozen representations of our adaptive methods.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “Rather than focusing on a simple single factor classification task, we evaluate the usefulness of disentangled representations on abstract visual reasoning tasks that challenge the current capabilities of state-of-the-art deep neural networks [30, 65].”
How Powerful are Graph Neural Networks? (2018, background): “Many GNN variants with different neighborhood aggregation and graph-level pooling schemes have been proposed (Scarselli et al., 2009b; Battaglia et al., 2016; Defferrard et al., 2016; Duvenaud et al., 2015; Hamilton et al., 2017a; Kearnes et al., 2016; Kipf & Welling, 2017; Li et al., 2016; Velickov”
Meta-Learning in Neural Networks: A Survey (2020, background): “Abstract Reasoning A recent goal in deep learning research is to develop models that go beyond simple perception tasks to solving more abstract reasoning problems such as IQ tests in the form of Raven’s Progressive Matrices (RPMs) [274].”
Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations (2018, methodology): “Apart from this difference in input feature dimensionality (64 in our VAE case vs 512 in the default CNN-encoder case), the entire WReN architecture is identical to the one used in [1].”
Learning to Make Analogies by Contrasting Abstract Relational Structure (2019, methodology): “Finally, we explore the extent to which LABC improves generalistion for different various different network architectures, taking as a guide the models considered in Barrett et al. (2018).”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “(Santoro et al., 2018) build a challenging dataset for solving the problem of abstract reasoning on the visual domain with some task such as interpolation or extrapolation.”
RAVEN: A Dataset for Relational and Analogical Visual REasoNing (2019, methodology): “In summary, we test a simple sequential learning model (LSTM), a CNN backbone with an MLP head (CNN), a ResNet-based [14] image classifier (ResNet), the recent relational WReN [3], and all these models augmented with the proposed DRT.”
Deep Nets: What have They Ever Done for Vision? (2018, background): “A recent example is when researchers (Santoro et al. 2018) tried to train standard Deep Nets to do IQ tests.”
Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, methodology): “While humans have strong abstract reasoning abilities, neural networks struggle with these problems [37, 38, 70, 72, 86].”
The Notorious Difficulty of Comparing Human and Machine Perception (2020, background): “In order to test if DNNs have similar cognitive abilities as humans, a number of studies test DNNs on abstract (visual) reasoning tasks (Barrett, Hill, Santoro, Morcos, & Lillicrap, 2018; Santoro et al., 2017; Villalobos et al., n.d.; Wu, Zhang, & Du, 2019; Yan & Zhou, 2017).”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Similarly, Santoro et al. (2018) show that predicting a “meta-target”—an abstract label encoding some task structure—improves learning of a relational reasoning task.”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, neutral): “tion [2], [7], [8], [9], [10], [11], [12].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, neutral): “2x2 3x3 Model Acc Grid Grid L-R U-D O-IC O-IG WReNDRT (Santoro et al., 2018) 15.”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, background): “Recently, the deep learning community started to tackle complex visual reasoning problems such as relationship detection [26], object recognition [9], abstract reasoning [33] or visual causality [25], while more theoretical work attempt to formalize relational reasoning [7].”