Efficient inference in occlusion-aware generative models of images

By Jonathan Huang, Kevin P. Murphy. 2015

44 citations recorded by the source corpus.

We present a generative model of images based on layering, in which image layers are individually generated, then composited from front to back. We are thus able to factor the appearance of an image into the appearance of individual objects within the image --- and additionally for each individual object, we can factor content from pose. Unlike prior work on layered models, we learn a shape prior for each object/layer, allowing the model to tease out which object is in front by looking for a consistent shape, without needing access to motion cues or any labeled data. We show that ordinary stochastic gradient variational bayes (SGVB), which optimizes our fully differentiable lower-bound on the log-likelihood, is sufficient to learn an interpretable representation of images. Finally we present experiments demonstrating the effectiveness of the model for inferring foreground and background objects in images.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “There has been recent work [8, 11, 25, 18, 41, 12, 16] in using deep generative models to disentangle multiple factors of variations (objects, pose, location, etc) from pixel data.”
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (2016, neutral): “Similar to our work is also Huang & Murphy (2015), however they assume a ﬁxed number of objects.”
Tagger: Deep Unsupervised Perceptual Grouping (2016, background): “Many of the works in that area [14, 29, 27, 11] build upon Restricted Boltzmann Machines.”
Deep Successor Reinforcement Learning (2016, background): “In order to scale up our approach to more expressive environments, it will be crucial to combine various deep generative and self-supervised models [6, 9, 10, 12, 16, 25, 27, 40]”
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation (2016, background): “Several works fall into this group (Roux et al., 2011; Huang & Murphy, 2015; Eslami et al., 2016).”
Robust Image Completion via Deep Feature Transformations (2019, background): “Unlike pooling layers, where the receptive fields are fixed and local, the spatial transformer provides a dynamic mechanism for producing an appropriate transformation for each input sample [39], [40].”
Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation (2019, unknown): “Layered representations have been studied for years in the computer vision community, and several methods which handle occlusions have been proposed (Wang & Adelson, 1994; Williams & Titsias, 2004; Le Roux et al., 2011; Huang & Murphy, 2016; Moreno et al., 2016; Wu et al., 2017).”
Factorized Variational Autoencoders for Modeling Audience Reactions to Movies (2017, background): “Building hierarchical Bayesian models with VAEs has until recently been elusive – one recent example is image generation with multiple layers [16].”
One-Shot Generalization in Deep Generative Models (2016, methodology): “The above sequential generative model and inference is a generalization of existing models such as DRAW (Gregor et al., 2015) , composited VAEs (Huang & Murphy, 2015) and AIR (Eslami et al., 2016).”
Unsupervised part representation by Flow Capsules (2020, background): “More closely related to ﬂow capsules, (Huang & Murphy, 2016) learn a generative model for shape within a layered image model, including depth ordering and occlusion.”
Unsupervised Discovery of Parts, Structure, and Dynamics (2019, background): “…and generative deep representation learning has been making progress along this direction: there have been models that efﬁciently explain multiple objects in a scene (Huang & Murphy, 2015; Eslami et al., 2016), some simultaneously learning an interpretable representation (Chen et al., 2016).”