Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

By S. M. Ali Eslami, Nicolas Manfred Otto Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, Geoffrey E. Hinton. 2016

584 citations recorded by the source corpus.

We present a framework for efficient inference in structured image models that explicitly reason about objects. We achieve this by performing probabilistic inference using a recurrent neural network that attends to scene elements and processes them one at a time. Crucially, the model itself learns to choose the appropriate number of inference steps. We use this scheme to learn to perform inference in partially specified 2D models (variable-sized variational auto-encoders) and fully specified 3D models (probabilistic renderers). We show that such models learn to identify multiple objects - counting, locating and classifying the elements of a scene - without any supervision, e.g., decomposing 3D images with various numbers of objects in a single forward pass of a neural network. We further show that the networks produce accurate inferences when compared to supervised counterparts, and that their structure leads to improved generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “[26] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al.”
One-Shot Generalization in Deep Generative Models (2016, methodology): “Models such as DRAW (Gregor et al., 2015), composited variational auto-encoders (Huang & Murphy, 2015) and AIR (Eslami et al., 2016) are existing models in this class, and we will develop a general class of sequential generative models that incorporates these and other latent variable models and var”
Recent advances in convolutional neural networks (2015, methodology): “In this section, we introduce some fast processing methods of CNNs.”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “In a similar vein, some authors have devised architectures that iteratively generate an image by rendering one object at a time [25,26], which also obliges the system to learn representations that comprise separately represented discrete entities.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “Even more recent approaches, such as SPAIR (Crawford and Pineau, 2019), SPACE (Lin et al., 2020), and SCALOR (Jiang et al., 2020), expand on this by incorporating explicit features for the presence of an object and its bounding box into each spatial slot.”
Towards causal generative scene models via competition of experts (2020, background): “Sequential models Our work is closely related to sequential or recurrent approaches to image decomposition and generation (Mnih et al., 2014; Gregor et al., 2015; Eslami et al., 2016; Kosiorek et al., 2018; Yuan et al., 2019).”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, background): “The Attend-Infer-Repeat (AIR) (Eslami et al., 2016) framework uses a recurrent neural network to attend to different objects in a scene and each object is sequentially processed one”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Finally, while this paper focuses on evaluating the benefits of disentangled features, these are complementary to recent work that focuses on the unsupervised “disentangling” of images into compositional primitives given by object-like representations [17, 23, 24, 22, 60, 74, 75].”
Multi-Object Representation Learning with Iterative Variational Inference (2019, background): “In line with recent advances (Burgess et al., 2019; van Steenkiste et al., 2018; Eslami et al., 2016), we maintain that discovery of objects in a scene should be considered a crucial aspect of representation learning, rather than treated as a separate problem.”
MONet: Unsupervised Scene Decomposition and Representation (2019, methodology): “The Attend-Infer-Repeat framework [Eslami et al., 2016] is the closest to our work in spirit.”
Generalization in anti-causal learning (2018, methodology): “In computer vision, 3D render engines have successfully been used in a line of research known as vision as inverse graphics to invert the physical process that produced an image (Loper and Black, 2014; Eslami et al., 2016).”
Relational inductive biases, deep learning, and graph networks (2018, background): “Or, it might be possible to use a separate learned mechanism to infer entities from an unstructured signal (Luong et al., 2015; Mnih et al., 2014; Eslami et al., 2016; van Steenkiste et al., 2018).”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “nd generate walking pedestrians without any supervision. Code for the implementation on the MNIST dataset2 and the results video3 are available online. 2Attend, Infer, Repeat (AIR) AIR, introduced by Eslami et al., 2016, is a structured variational auto-encoder (VAE) capable of decomposing a static ”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “Other unsupervised approaches rely on attention to group together parts of the visual scene corresponding to objects (Eslami et al., 2016; Gregor et al., 2015).”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “LSTM Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) have been shown to sequentially attend to objects (Eslami et al., 2016), so it is interesting to test whether a LSTM is well-suited for modeling object interactions, when the object states are explicitly given as input.”
Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks (2019, methodology): “A recent approach called Attend, Infer, Repeat (AIR) (Eslami et al. 2016) has shown promise at unsupervised object detection, but (as we show empirically) has difficulty scaling up to handle large images containing many objects.”