Discovering objects and their relations from entangled scene representations

By David Raposo, Adam Santoro, David G. T. Barrett, Razvan Pascanu, Timothy P. Lillicrap, Peter W. Battaglia. 2017

77 citations recorded by the source corpus.

Our world can be succinctly and compactly described as structured scenes of objects and relations. A typical room, for example, contains salient objects such as tables, chairs and books, and these objects typically relate to each other by their underlying causes and semantics. This gives rise to correlated features, such as position, function and shape. Humans exploit knowledge of objects and their relations for learning a wide spectrum of tasks, and more generally when learning the structure underlying observed data. In this work, we introduce relation networks (RNs) - a general purpose neural network architecture for object-relation reasoning. We show that RNs are capable of learning object relations from scene description data. Furthermore, we show that RNs can act as a bottleneck that induces the factorization of objects from entangled scene description inputs, and from distributed deep representations of scene images provided by a variational autoencoder. The model can also be used in conjunction with differentiable memory mechanisms for implicit relation discovery in one-shot learning tasks. Our results suggest that relation networks are a potentially powerful architecture for solving a variety of problems that require object relation reasoning.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Relation Networks for Object Detection (2017, methodology): “Attention modules in NLP and physical system modeling Attention modules have recently been successfully applied in the NLP field [20, 21, 6, 56] and in physical system modeling [2, 57, 31, 45, 13, 42].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “In the near future, we plan to also adapt the technique of multimodal compact bilinear pooling (Fukui et al., 2016), neural module networks (Andreas et al., 2016a,b) and potentially also relation networks (Raposo et al., 2017) to the ICA task, and upload implementations to the GitHub repository.”
A simple neural network module for relational reasoning (2017, background): “RNs are architectures whose computations focus explicitly on relational reasoning [18].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Finally, while this paper focuses on evaluating the benefits of disentangled features, these are complementary to recent work that focuses on the unsupervised “disentangling” of images into compositional primitives given by object-like representations [17, 23, 24, 22, 60, 74, 75].”
Relational inductive biases, deep learning, and graph networks (2018, background): “Gated Graph Sequence Neural Networks (GGS-NN) (Li et al., 2016) use a slightly generalized formulation where each edge has an attached type t k ∈ { 1 , .., T } , and the updates are: φ e (( e k , t k ) , v r k , v s k , u ) := f e ( e k , v s k ) = NN e,t k ( v s k ) These updates are applied recurr”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Recent work has shown empirically that concepts are discovered sequentially, even if they are simultaneously present in the data [17], and the behavior is predicted analytically for deep linear models A –”
Measuring the tendency of CNNs to Learn Surface Statistical Regularities (2017, methodology): “In [16] a variational setup is used to learn visual concepts and [32] aims to learn abstract relations be-tween objects in natural scene images.”
Graph networks as learnable physics engines for inference and control (2018, neutral): “…et al., 2013; Li et al., 2015; Henaff et al., 2015; Du-venaud et al., 2015; Dai et al., 2016; Defferrard et al., 2016; Niepert et al., 2016; Kipf & Welling, 2016; Battaglia et al., 2016; Watters et al., 2017; Raposo et al., 2017; Santoro et al., 2017; Bronstein et al., 2017; Gilmer et al., 2017).”
Relational recurrent neural networks (2018, background): “In Relation Networks [15–17] entities are obtained by exploiting spatial locality in the input image, and the model focuses on computing binary relations between each entity pair.”
Measuring abstract reasoning in neural networks (2018, methodology): “We designed a novel variant of the Relation Network (Santoro et al., 2017; Raposo et al., 2017), a neural network with speciﬁc structure designed to encourage relation-level comparisons and reasoning.”
Relational Forward Models for Multi-Agent Learning (2018, positive): “…entities and a rich relational structure, like physics simulation, multi-object scenes, visual question answering and motion-capture data (Battaglia et al., 2016; Raposo et al., 2017; Santoro et al., 2017; Gilmer et al., 2017; Watters et al., 2017; Kipf et al., 2018; Zambaldi et al., 2018).”
Differentiable Scene Graphs (2019, background): “Extracting scene graphs from images provides a semantic representation that can later be used for reasoning, question answering [40], and image retrieval [15, 31].”
Neural Production Systems (2021, methodology): “Only in the past few years have deep learning vision researchers developed methods for object-centric representation (Le Roux et al., 2011; Eslami et al., 2016; Greff et al., 2016; Raposo et al., 2017; Van Steenkiste et al., 2018; Kosiorek et al., 2018; Engelcke et al., 2019; Burgess et al., 2019; G”
Graph Neural Networks: A Review of Methods and Applications (2018, methodology): “[27] proposed the graph network (GN) which unified the MPNN and NLNN methods as well as many other variants like Interaction Networks [4], [64], Neural Physics Engine [65], CommNet [66], structure2vec [7], [67], GGNN [45], Relation Network [68], [69], Deep Sets [70] and Point Net [71].”
Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study (2017, methodology): “…1985) to the recent ﬂurry of episodic memory architectures (Blun-dell et al., 2016; Graves et al., 2016), and to apply techniques from the semantic cognition literature (Lamberts & Shanks, 2013) to recent models of concept formation (Hig-gins et al., 2016; Gregor et al., 2016; Raposo et al., 2017).”
Pixels to Graphs by Associative Embedding (2017, background): “Visual relationship detection has drawn much recent attention [18, 28, 27, 2, 17, 19, 22, 23].”