Towards causal generative scene models via competition of experts

By Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, Bernhard Schölkopf. 2020

54 citations recorded by the source corpus.

Learning how to model complex scenes in a modular way with recombinable components is a pre-requisite for higher-order reasoning and acting in the physical world. However, current generative models lack the ability to capture the inherently compositional and layered nature of visual scenes. While recent work has made progress towards unsupervised learning of object-based scene representations, most models still maintain a global representation space (i.e., objects are not explicitly separated), and cannot generate scenes with novel object arrangement and depth ordering. Here, we present an alternative approach which uses an inductive bias encouraging modularity by training an ensemble of generative models (experts). During training, experts compete for explaining parts of a scene, and thus specialise on different object classes, with objects being identified as parts that re-occur across multiple scenes. Our model allows for controllable sampling of individual objects and recombination of experts in physically plausible ways. In contrast to other methods, depth layering and occlusion are handled correctly, moving this approach closer to a causal generative scene model. Experiments on simple toy data qualitatively demonstrate the conceptual advantages of the proposed approach.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[67] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf.”
Toward Causal Representation Learning (2021, background): “Note that the problem of object-centric representation learning [11], [40], [84], [87], [88], [138], [155], [160], [255], [262] can also be considered a special case of disentangled factorization as discussed here.”
Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
On the Binding Problem in Artificial Neural Networks (2020, background): “For example, in MONet (Burgess et al., 2019), GENESIS (Engelcke et al., 2019), and ECON (von Kügelgen et al., 2020) a recurrent neural network is trained to directly support the learning of object representations by outputting a mask that focuses on different objects at each step14.”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, unknown): “This is in line with a recently proposed [105] shift towards causal representation learning [40, 76, 85, 87, 106, 107, 112, 123, 127], motivated by the fact that many underlying variables of interest may not be independent but causally related to each other.”
Counterfactual Generative Networks (2021, methodology): “Lastly, methods for causal generative modeling utilizing competing experts (von Kügelgen et al., 2020) have been demonstrated on toy datasets only.”