Picture: A probabilistic programming language for scene perception

By Tejas D. Kulkarni, Pushmeet Kohli, Josh Tenenbaum, Vikash K. Mansinghka. 2015

211 citations recorded by the source corpus.

Recent progress on probabilistic modeling and statistical learning, coupled with the availability of large training datasets, has led to remarkable progress in computer vision. Generative probabilistic models, or “analysis-by-synthesis” approaches, can capture rich scene structure but have been less widely applied than their discriminative counterparts, as they often require considerable problem-specific engineering in modeling and inference, and inference is typically seen as requiring slow, hypothesize-and-test Monte Carlo methods. Here we present Picture, a probabilistic programming language for scene understanding that allows researchers to express complex generative vision models, while automatically solving them using fast general-purpose inference machinery. Picture provides a stochastic scene language that can express generative models for arbitrary 2D/3D scenes, as well as a hierarchy of representation layers for comparing scene hypotheses with observed images by matching not simply pixels, but also more abstract features (e.g., contours, deep neural network activations). Inference can flexibly integrate advanced Monte Carlo strategies with fast bottom-up data-driven methods. Thus both representations and inference strategies can build directly on progress in discriminatively trained systems to make generative vision more robust and efficient. We use Picture to write programs for 3D face analysis, 3D human pose estimation, and 3D object reconstruction - each competitive with specially engineered baselines.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

An Introduction to Probabilistic Programming (2018, methodology): “An example of this “vision as inverse graphics” idea (Kulkarni et al., 2015b) appearing first in Mansinghka et al. (2013) and then subsequently in Le et al. (2017b,a) took the image Y to be a Captcha image and the scene description X to include the obscured string.”
Deep Convolutional Inverse Graphics Network (2015, background): “Recent work in inverse graphics [15, 12, 11] follows a general strategy of defining a probabilistic with latent parameters, then using an inference algorithm to find the most appropriate set of latent parameters given the observations.”
Masked Autoregressive Flow for Density Estimation (2017, background): “Density estimators can learn effective proposals for importance sampling [22] or sequential Monte Carlo [8, 21]; such proposals can be used in probabilistic programming environments to speed up inference [15, 16].”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “, 2013) and incorporated deep neural networks to help make inference more tractable (Kulkarni et al., 2015; Romaszko et al., 2017).”
Towards causal generative scene models via competition of experts (2020, methodology): “Probabilistic programming is often advocated as a means to automatically compile this inference task; for instance, PICTURE has been proposed by Kulkarni et al. (2015), and combinations with deep learning have been explored by Wu et al. (2017).”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “…strength, but automatically inferring from visual data what physical laws and object properties are present requires more work in inverse graphics (Chen et al., 2016; Kulkarni et al., 2014; 2015a;b; Whitney et al., 2016) and physics-based visual understanding (Brand, 1997; Wu et al., 2015; 2016).”
End-to-End Recovery of Human Shape and Pose (2017, background): “[19] use a generative model of body shape and pose with a probabilistic programming framework to estimate body pose from single images.”
Unsupervised Learning of 3D Structure from Images (2016, methodology): “f of the data volume. Right: 100th iteration of the MCMC chain. Best viewed on a screen. Videos of these samples can be seen at https://goo.gl/9hCkxs. data is then augmented with random translations ([0;20] pixels) and rotations ([ ˇ;ˇ] radians) that are procedurally applied during training. ShapeNe”
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (2016, methodology): “For instance, in PICTURE [11], the authors use a deep network to distill the results of slow MCMC, speeding up predictions at test-time.”
Learning Disentangled Representations with Semi-Supervised Deep Generative Models (2017, background): “deep generative models [8, 17, 20, 21], which impose very few assumptions on the structure of the model, instead employing neural networks as flexible function approximators that can be used to train a conditional distribution on the data, rather than specify it by hand.”
Single Image 3D Interpreter Network (2016, methodology): “obust to imperfect detection results and can be jointly learned with keypoint estimators. Our work also connects to the traditional eld of vision as inverse graphics [14,21] and analysis by synthesis [58,20,5,52], as we use neural nets to decode latent 3D structure from images, and use a projection ”
Image Generation from Scene Graphs (2018, background): “have investigated text to 3D scene generation [4, 5]; other approaches to image synthesis include stochastic grammars [20], probabalistic programming [27], inverse graphics [28], neural de-rendering [55], and generative ConvNets [56].”
Neural Scene De-rendering (2017, methodology): “Our framework also relates to the field of “vision as inverse graphics”, analysis-by-synthesis, or generative models with data-driven proposals [36, 37, 30, 20, 34, 16], as we are incorporating a graphics engine as a black-box synthesizer.”
VirtualWorlds as Proxy for Multi-object Tracking Analysis (2016, background): “From early on, computer vision researchers leveraged 3D computer simulations to model articulated objects including human shape [8], face, and hand appearance [9], or even for scene interpretation and vision as inverse graphics [10, 11, 12].”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, background): “…and learning feed-forward or recurrent procedures for image generation (LeCun et al., 2015; Hinton & Salakhutdinov, 2006; Goodfellow et al., 2014; Ackley et al., 1987; Kingma & Welling, 2013; Oord et al., 2016; Kulkarni et al., 2015b; Eslami et al., 2016; Reed et al., 2017; Gregor et al., 2015).”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, methodology): “More recently, some applications of general purpose inverse graphics solutions using probabilistic programming languages have been reported (Mansinghka et al. 2013; Loper et al. 2014; Kulkarni et al. 2015).”