Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs

By Vikash K. Mansinghka, Tejas D. Kulkarni, Yura N. Perov, Josh Tenenbaum. 2013

114 citations recorded by the source corpus.

The idea of computer vision as the Bayesian inverse problem to computer graphics has a long history and an appealing elegance, but it has proved difficult to directly implement. Instead, most vision tasks are approached via complex bottom-up processing pipelines. Here we show that it is possible to write short, simple probabilistic graphics programs that define flexible generative models and to automatically invert them to interpret real-world images. Generative probabilistic graphics programs (GPGP) consist of a stochastic scene generator, a renderer based on graphics software, a stochastic likelihood model linking the renderer's output and the data, and latent variables that adjust the fidelity of the renderer and the tolerance of the likelihood. Representations and algorithms from computer graphics are used as the deterministic backbone for highly approximate and stochastic generative models. This formulation combines probabilistic programming, computer graphics, and approximate Bayesian computation, and depends only on general-purpose, automatic inference techniques. We describe two applications: reading sequences of degraded and adversarially obscured characters, and inferring 3D road models from vehicle-mounted camera images. Each of the probabilistic graphics programs we present relies on under 20 lines of probabilistic code, and yields accurate, approximately Bayesian inferences about real-world images.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

An Introduction to Probabilistic Programming (2018, methodology): “An example of this “vision as inverse graphics” idea (Kulkarni et al., 2015b) appearing first in Mansinghka et al. (2013) and then subsequently in Le et al. (2017b,a) took the image Y to be a Captcha image and the scene description X to include the obscured string.”
Deep Convolutional Inverse Graphics Network (2015, methodology): “Recent work in inverse graphics [11, 19, 18, 14] follows a general strategy of ﬁrst deﬁning a probabilistic or non-probabilistic model with latent parameters, followed by an inference or optimization algorithms to ﬁnd the most appropriate set of latent parameters given observations.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “For example, they have enabled the use of symbolic graphic renderers as forward models (Mansinghka et al., 2013) and incorporated deep neural networks to help make inference more tractable (Kulkarni et al., 2015; Romaszko et al., 2017).”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, background): “While some prior works on hierarchical Bayesian models have proposed to handle basic image recognition tasks, the complexity of these tasks does not yet approach the kinds of complex image recognition problems that can be solved by discriminatively trained deep networks, such as the miniImageNet exp”
Unsupervised Learning of 3D Structure from Images (2016, methodology): “In our experiments, context is either 1) nothing, 2) an object class label, or 3) one or more views of the scene from different cameras.”
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (2016, background): “These models range from highly specified [17, 16], to partially specified [28, 24, 25], to largely unspecified [22].”
Picture: A probabilistic programming language for scene perception (2015, methodology): “global or local blur variables in GPGP [31]), and (b) to soak up model mismatch between the true scene rendering JD and the hypothesized rendering JR.”
VirtualWorlds as Proxy for Multi-object Tracking Analysis (2016, background): “From early on, computer vision researchers leveraged 3D computer simulations to model articulated objects including human shape [8], face, and hand appearance [9], or even for scene interpretation and vision as inverse graphics [10, 11, 12].”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, background): “The idea of inverting simulators to interpret images has been explored extensively in recent years (Nair et al., 2008; Paysan et al., 2009; Mansinghka et al., 2013; Loper & Black, 2014; Kulkarni et al., 2015a; Jampani et al., 2015).”
Inference Compilation and Universal Probabilistic Programming (2016, methodology): “In our second, we demonstrate Captcha solving via probabilistic inference (Mansinghka et al., 2013).”
Probabilistic machine learning and artificial intelligence (2015, methodology): “Another application of probabilistic programming implements a computer vision system as the inverse of a computer graphics program 50 .”
Displets: Resolving stereo ambiguities using object knowledge (2015, methodology): “In contrast to [38], our generative process produces disparity maps from CAD models using the camera intrinsics.”
The informed sampler: A discriminative approach to Bayesian inference in generative computer vision models (2014, methodology): “This view was advocated since the late 1970’ies (Horn, 1977; Grenander, 1976; Zhu and Mumford, 1997; Mumford and Desolneux, 2010; Mansinghka et al., 2013; Yuille and Kersten, 2006).”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “[24, 32] use such probabilistic programs to invert the generative process and infer a program given an image, while we”
Joint 3D Object and Layout Inference from a Single RGB-D Image (2015, methodology): “Instead, we sample proposals directly from the unary distributions specified in Section 3.1 using Metropolis-Hastings [9, 26], leveraging the power of our 3D models in a generative manner.”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “[28] propose a general inverse rendering technique by using a probabilistic graphics model to infer scene parameters from observations.”
Probabilistic programming with programmable inference (2018, methodology): “Inference metaprogramming in Venture has been used to solve problems in diverse fields, including probabilistic robotics [10, 36], computer vision [30], inverse planning [9], geophysics [41], and data science [6, 29, 43].”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “Learning with Simulators: Methods in Approximate Bayesian Inference have looked into inferring the parameters of a simulator that generate a particular data point [45,35].”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “Mansinghka et al. (2013) propose a inverse-graphics inspired method employing generative probabilistic graphics programs (GPGP) to infer roads in images taken from vehicle mounted cameras.”