Vision-as-Inverse-Graphics: Obtaining a Rich 3D Explanation of a Scene from a Single Image

By Lukasz Romaszko, Christopher K. I. Williams, Pol Moreno, Pushmeet Kohli. 2017

46 citations recorded by the source corpus.

We develop an inverse graphics approach to the problem of scene understanding, obtaining a rich representation that includes descriptions of the objects in the scene and their spatial layout, as well as global latent variables like the camera parameters and lighting. The framework's stages include object detection, the prediction of the camera and lighting variables, and prediction of object-specific variables (shape, appearance and pose). This acts like the encoder of an autoencoder, with graphics rendering as the decoder Importantly the scene representation is interpretable and is of variable dimension to match the detected number of objects plus the global variables. For the prediction of the camera latent variables we introduce a novel architecture termed Probabilistic HoughNets (PHNs), which provides a principled approach to combining information from multiple detections. We demonstrate the quality of the reconstructions obtained quantitatively on synthetic data, and qualitatively on real scenes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2013) and incorporated deep neural networks to help make inference more tractable (Kulkarni et al., 2015; Romaszko et al., 2017).”
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare (2018, methodology): “Recently there has been a re-emergence of the inversegraphics approach [11, 27, 44, 60, 51, 24, 25, 34], in which an efficient, discriminative bottom-up method like a convolutional network is used to cut down on the search space.”
Automatic Posterior Transformation for Likelihood-Free Inference (2019, background): “This scenario is common in neuroscience (Paninski & Cunningham, 2017), physics (Brehmer et al., 2018), systems biology (Clarke et al., 2008) and inverse graphics (Romaszko et al., 2017).”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “while in computer vision, the prior is often overlooked, or inaccurately considered to be a uniform distribution, making MAP equivalent to Maximum Likelihood (ML), i.e., nding G that maximizes p(IjG) [20]. We propose the rst method to explicitly approximate the MAP inference by devising an explicit ”
Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows (2018, background): “For instance, inferring the parameters of physical theories given particle collisions can be the means for discovering new physics [11], and inferring the scene parameters of a graphics engine given a natural image can be the basis for computer vision [66, 87].”
Geometrical Insights for Implicit Generative Modeling (2017, methodology): “This approach is well illustrated by the inverse graphics approach to computer vision [31, 30, 43].”
The Multi-Entity Variational Autoencoder (2018, background): “Another related strand of work focuses on explicit representations of multiple objects in visual scenes, making use of graphics engines as a known decoding function [17, 11].”