SCALOR: Generative World Models with Scalable Object Representations

By Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, Sungjin Ahn. 2019

154 citations recorded by the source corpus.

Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with a few objects. In this paper, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatially parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous state-of-the-art models. Additionally, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds. Importantly, SCALOR is the ﬁrst unsupervised object representation model shown to work for natural scenes containing several tens of moving objects.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “Our attention-based routing procedure could also be employed in conjunction with patch-based decoders, used in architectures such as AIR [26], SQAIR [40], and related approaches [41–44], as an alternative to the typically employed autoregressive encoder [26, 40].”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2020), and SCALOR (Jiang et al., 2020), expand on this by incorporating explicit features for the presence of an object and its bounding box into each spatial slot.”
Generative Neurosymbolic Machines (2020, background): “For example, in object-centric latent variable models [13, 30, 10, 24], a univariate Gaussian distribution p(zwhere x ) = N (μx, σx) can be introduced to define a symbolic prior on the x-coordinate of an object in an image.”
Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking (2019, methodology): “…dealing with complex, dynamic backgrounds; past work in this area has generally preprocessed videos with an off-the-shelf background subtractor, largely avoiding the problem of modeling the background (though see concurrent work in (Jiang et al. 2019) which makes signiﬁcant progress on this issue).”
End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning (2019, background): “To the best of our knowledge, there are no successful implementations of any such end-to-end approaches for multi-object tracking from vision beyond generative modelling works [14, 26, 11] which work only on relatively simple datasets, and Frossard and Urtasun [6], which relies on costly LIDAR senso”
Hierarchical Relational Inference (2020, methodology): “Neural Relational Inference To infer relations between object representations, we will make use of NRI Kipf et al. (2018), which learns explicit relations.”
Opening up Open World Tracking (2021, methodology): “Methods by [45, 19, 39] learn to decompose scenes into a set of object instances based on the input video sequence in an unsupervised manner.”