Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects

By Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh. 2018

277 citations recorded by the source corpus.

We present Sequential Attend, Infer, Repeat (SQAIR), an interpretable deep generative model for image sequences. It can reliably discover and track objects through the sequence; it can also conditionally generate future frames, thereby simulating expected motion of objects. This is achieved by explicitly encoding object numbers, locations and appearances in the latent variables of the model. SQAIR retains all strengths of its predecessor, Attend, Infer, Repeat (AIR, Eslami et. al. 2016), including unsupervised learning, made possible by inductive biases present in the model structure. We use a moving multi-\textsc{mnist} dataset to show limitations of AIR in detecting overlapping or partially occluded objects, and show how \textsc{sqair} overcomes them by leveraging temporal consistency of objects. Finally, we also apply SQAIR to real-world pedestrian CCTV data, where it learns to reliably detect, track and generate walking pedestrians with no supervision.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “Our attention-based routing procedure could also be employed in conjunction with patch-based decoders, used in architectures such as AIR [26], SQAIR [40], and related approaches [41–44], as an alternative to the typically employed autoregressive encoder [26, 40].”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “AIR (Eslami et al., 2016) and its sequential extension SQAIR (Kosiorek et al., 2018) deploy a similar strategy for an unsupervised learning task with the purpose of extracting object representations.”
Towards causal generative scene models via competition of experts (2020, background): “Sequential models Our work is closely related to sequential or recurrent approaches to image decomposition and generation (Mnih et al., 2014; Gregor et al., 2015; Eslami et al., 2016; Kosiorek et al., 2018; Yuan et al., 2019).”
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition (2020, background): “In a related line of research, AIR has also recently been extended to track objects in a sequence of images (Kosiorek et al., 2018; Crawford & Pineau, 2020; Jiang et al., 2020).”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “The recent sequential AIR (attend, infer, repeat) model of Kosiorek et al. (2018) uses a variable-sized, object-oriented latent state space rather than a ﬁxed-dimensional unstructured latent vector.”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “This additive interaction is restrictive, and as a consequence, neither AIR nor its successor SQAIR [Kosiorek et al., 2018] can model occluded objects or background pixels.”
Stacked Capsule Autoencoders (2019, background): “ion in computer vision. The ﬁrst one is based on representation learning and typically requires discovering clusters or learning a classiﬁer on top of the learned representation. Eslami et al., 2016; Kosiorek et al., 2018 use an iterative procedure to infer a variable number of latent variables, one”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “image classification and captioning [12, 13, 14, 15, 16, 17, 18, 19, 20], text classification [21, 22], generative models [6, 23, 24], object tracking [25], and reinforcement learning [26, 27, 28].”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, background): “Attend-Infer-Repeat (Eslami et al. [13]) and its sequential counterpart in Kosiorek et al. [14] use spatial attention to partition scenes into objects.”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, positive): “[19] proposed the Sequential Attend-Infer-Repeat (SQAIR), which extends the AIR model and is very similar to our work.”
Neural Production Systems (2021, background): “Only in the past few years have deep learning vision researchers developed methods for object-centric representation (Le Roux et al., 2011; Eslami et al., 2016; Greff et al., 2016; Raposo et al., 2017; Van Steenkiste et al., 2018; Kosiorek et al., 2018; Engelcke et al., 2019; Burgess et al., 2019; G”
Transformation-based Adversarial Video Prediction on Large-Scale Data (2020, unknown): “Object-centric representations have also been proposed [38, 83].”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, background): “A few approaches explicitly model interactions between objects inferred from an observed scene [14, 27, 76].”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “Such learned approximations of object interactions have been successfully employed in object manipulation [15] and tracking [9, 21].”
On the Limitations of Representing Functions on Sets (2019, background): “…of machine learning problems can naturally be formulated in terms of sets; e.g. parsing a scene composed of a set of objects (Eslami et al., 2016; Kosiorek et al., 2018), making predictions from a set of points forming a 3D point cloud (Qi et al., 2017a;b), or training a set of agents in…”
Scalable Object-Oriented Sequential Generative Models (2019, background): “SQAIR (Kosiorek et al., 2018) extends the ideas proposed in AIR to temporal sequences.”
A Perspective on Objects and Systematic Generalization in Model-Based RL (2019, background): “A downside is that objects are processed in an iterative fashion, which may make it more challenging to reason about multiple objects simultaneously (Kosiorek et al., 2018).”