A simple neural network module for relational reasoning

By Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, Timothy P. Lillicrap. 2017

1,703 citations recorded by the source corpus.

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Graph Attention Networks (2017, background): “Moreover, our approach of sharing a neural network computation across edges is reminiscent of the formulation of relational networks (Santoro et al., 2017) and VAIN (Hoshen, 2017), wherein relations between objects or agents are aggregated pair-wise, by employing a shared mechanism.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Santoro et al. (2017), Norcliffe-Brown et al. (2018), and Cadene et al. (2019) explore using an attention module to implicitly model the relations between objects in the image.”
Relation Networks for Object Detection (2017, methodology): “Attention modules in NLP and physical system modeling Attention modules have recently been successfully applied in the NLP field [20, 21, 6, 56] and in physical system modeling [2, 57, 31, 45, 13, 42].”
Non-local Neural Networks (2017, methodology): “Concatenation is used by the pairwise function in Relation Networks [40] for visual reasoning.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
VAIN: Attentional Multi-agent Predictive Modeling (2017, neutral): “Santoro et al. [30] discovered that an architecture nearly identical to Interaction Nets achieves excellent performance on the CLEVR dataset [31].”
Neural Logic Machines (2019, background): “Santoro et al. (2017) implements a visual reasoning system based on “virtual” objects brought by receptive ﬁelds in CNNs. Wu et al. (2017) tackles the problem of deriving structured representation from raw pixel-level inputs.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “We make some changes to the training procedure of DSPN: • Instead of using a relation network (Santoro et al., 2017) – expanding the set into the set of all pairs ﬁrst – paired with FSPool, we skip the relation network entirely.”
Evaluating Theory of Mind in Question Answering (2018, methodology): “We report results using a single seed because we saw little randomness due to initialization; this is in accordance with the authors’ ﬁndings (Santoro et al., 2017).”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “Some have argued that for artiﬁcial agents to learn this complex, structured process, it is necessary to build in aspects of reasoning, such as compositionality (Hu et al. 2017; Johnson et al. 2017b) or relational computation (Santoro et al. 2017).”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…2017b; Perez et al., 2018; Hu et al., 2017; Mascharka et al., 2018; Hu et al., 2018; Suarez et al., 2018) and attention-or memory-based meth-ods (Santoro et al., 2017; Zhu et al., 2017; Hud-son and Manning, 2018; Tan and Bansal, 2018; Malinowski et al., 2018; Pavez et al., 2018; Yao et al.,…”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “0 [23], CLEVR [32], Visual7w [81], Visual Genome [41], COCO-QA [57], and models [33, 60, 29, 31, 1, 67, 17, 37] have been used to measure the progress in understanding the interaction between vision and language modalities.”
Compositional Attention Networks for Machine Reasoning (2018, methodology): “Based on the analogy suggested by Santoro et al. (2017), according to which pixels are equivalent to objects and their pairwise interactions to relations, an RN layer attempts to grasp the induced graph between objects all at once in one shallow and broad layer.”
In Defense of Grid Features for Visual Question Answering (2020, methodology): “CLEVR [17]), simple methods based on grids [33, 31] have shown strong performance.”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In reinforcement learning, very few examples of symbolic (graphical [311] or relational [75,312] ) hybrid models exist, while in recommendation systems, for instance, explainable autoencoders are proposed [313] .”
Toward Causal Representation Learning (2021, background): “Machine learning models that incorporate or learn structural knowledge of an environment have been shown to be more efﬁcient and generalize better [9], [11], [15], [16], [27], [58], [77], [84], [85], [141], [157], [177], [181], [197], [211], [212], [244], [258], [272], [274].”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “[4], [41], NTN[47], [48], etc.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “The used methods may either be components that add additional capabilities, components that belong to the model architecture [75], [76], e.”