RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces

By Sébastien Ehrhardt, Oliver Groth, Áron Monszpart, Martin Engelcke, Ingmar Posner, Niloy J. Mitra, Andrea Vedaldi. 2020

60 citations recorded by the source corpus.

We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We find that RELATE is also amenable to physically realistic scene editing and that it significantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (street traffic scenes). In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “(2020) demonstrates how a (spatial) mixture model that combines the output of multiple deep neural networks is able to learn to generate images as compositions of individual objects and a background (see also Nguyen-Phuoc et al., 2020; Ehrhardt et al., 2020; Niemeyer and Geiger, 2020).”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, background): “Regarding this, there have been a number of recent approaches that can (only) generate 3D scene images via object-centric compositional rendering (Nguyen-Phuoc et al., 2020; van Steenkiste et al., 2020; Ehrhardt et al., 2020).”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, background): “In terms of what we consider to present the most opportunity for future work, we note that there are relatively few continuous optimization approaches which seek to learn structured, semantic representations from nonsemantic, high-dimensional data such as video or image data (exceptions include Caus”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Recent works therefore investigate how the synthesis process can be controlled at the object-level [3,4,7,18,19,26,45,86,90].”
Generative Adversarial Transformers (2021, background): “Most related to our work are certain GAN models for conditional and unconditional visual synthesis: A few meth-ods [21, 33, 54, 64] utilize multiple replicas of a generator to produce a set of image layers, that are then combined through alpha-composition.”