Deep Convolutional Inverse Graphics Network

By Tejas D. Kulkarni, William F. Whitney, Pushmeet Kohli, Josh Tenenbaum. 2015

951 citations recorded by the source corpus.

This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that aims to learn an interpretable representation of images, disentangled with respect to three-dimensional scene structure and viewing transformations such as depth rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm [10]. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative tests of the model's efficacy at learning a 3D rendering engine for varied object classes including faces and chairs.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “DC-IGN [7] extends this “clamping” idea to VAE and successfully learns graphics codes that can represent pose and light in 3D rendered images.”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017; Suter et al., 2019).”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “In terms of synthesis, various approaches based on the ConvNet have been proposed for synthesizing realistic static images [13], [14], [15], [16], [17].”
Unrolled Generative Adversarial Networks (2016, background): “…denoising autoencoders (Vincent et al., 2010), variational au-toencoders (Kingma & Welling, 2013; Rezende et al., 2014; Gregor et al., 2015; Kulkarni et al., 2015; Burda et al., 2015; Kingma et al., 2016), generative stochastic networks (Alain et al., 2015), diffusion probabilistic models…”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Game Engines as World Models : The concept of using TextWorld as a “game engine” to represent the world is broadly related to inverse graphics (Kulkarni et al., 2015) and inverse dynamics (Wu et al., 2017) where abstract visual or physical models are used for reasoning and future predictions.”
An Introduction to Probabilistic Programming (2018, methodology): “An example of this “vision as inverse graphics” idea (Kulkarni et al., 2015b) appearing first in Mansinghka et al. (2013) and then subsequently in Le et al. (2017b,a) took the image Y to be a Captcha image and the scene description X to include the obscured string.”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, methodology): “We consider two different network types for our model: Standard fully connected neural networks with up to three layers which work well for moderately sized images and are used for the planar and swing-up experiments; A deep convolutional network for the encoder in combination with an “up-convolutio”
Importance Weighted Autoencoders (2015, background): “VAEs have recently been successful at separating style and content (Kingma et al., 2014; Kulkarni et al., 2015) and at learning to “draw” images in a realistic manner (Gregor et al., 2015).”
Autoencoding beyond pixels using a learned similarity metric (2015, background): “Later, Kulkarni et al. (2015); Yan et al. (2015); Reed et al. (2015) have demonstrated encoder-decoder architectures with disentan-gled feature representations, but their training schemes rely on supervised information.”
Learning Image Representations Tied to Ego-Motion (2015, methodology): “Methods to learn representations with disentangled latent factors [11, 14] aim to sort properties like pose, illumination etc.”
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (2016, background): “These models can also be augmented semantic attributes [35], image captions [24], 3D data [5,17], spatial/temporal status [11,13,27] etc.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “To generate interpretable images with richer information, the VAE is extended to be conditioned on captions [29] and graphics code [30].”
Attribute2Image: Conditional Image Generation from Visual Attributes (2015, methodology): “In terms of generating realistic and novel images, there are a few recent work [8, 13, 23, 12, 6] that are relevant to ours.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Early approaches toward this goal includes DC-IGN (Kulkarni et al., 2015) and Spatial Transformers (Jaderberg et al.”
Tutorial on Variational Autoencoders (2016, background): “Ideally, we want to avoid deciding by hand what information each dimension of z encodes (although we may want to specify it by hand for some dimensions [4]).”