Inverse Graphics with Probabilistic CAD Models

By Tejas D. Kulkarni, Vikash K. Mansinghka, Pushmeet Kohli, Josh Tenenbaum. 2014

19 citations recorded by the source corpus.

Recently, multiple formulations of vision problems as probabilistic inversions of generative models based on computer graphics have been proposed. However, applications to 3D perception from natural images have focused on low-dimensional latent scenes, due to challenges in both modeling and inference. Accounting for the enormous variability in 3D object shape and 2D appearance via realistic generative models seems intractable, as does inverting even simple versions of the many-to-many computations that link 3D scenes to 2D images. This paper proposes and evaluates an approach that addresses key aspects of both these challenges. We show that it is possible to solve challenging, real-world 3D vision problems by approximate inference in generative models for images based on rendering the outputs of probabilistic CAD (PCAD) programs. Our PCAD object geometry priors generate deformable 3D meshes corresponding to plausible objects and apply affine transformations to place them in a scene. Image likelihoods are based on similarity in a feature space based on standard mid-level image representations from the vision literature. Our inference algorithm integrates single-site and locally blocked Metropolis-Hastings proposals, Hamiltonian Monte Carlo and discriminative data-driven proposals learned from training data generated from our models. We apply this approach to 3D human pose estimation and object shape reconstruction from single images, achieving quantitative and qualitative performance improvements over state-of-the-art baselines.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Flexible Neural Representation for Physics Prediction (2018, methodology): “More similar to our approach are inverse graphics methods that extract a lower dimensional physical representation from images that is used to predict physics [25, 26, 51, 50, 52, 53, 7, 49].”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “…strength, but automatically inferring from visual data what physical laws and object properties are present requires more work in inverse graphics (Chen et al., 2016; Kulkarni et al., 2014; 2015a;b; Whitney et al., 2016) and physics-based visual understanding (Brand, 1997; Wu et al., 2015; 2016).”
Unsupervised Learning of 3D Structure from Images (2016, methodology): “In our experiments, context is either 1) nothing, 2) an object class label, or 3) one or more views of the scene from different cameras.”
Disentangled Representations in Neural Models (2016, background): “Recent work in inverse graphics (Mansinghka et al. 2013, Kulkarni et al. (2014), Kulkarni, Kohli, et al. (2015)) follows a general strategy of defining a probabilistic with latent parameters, then using an inference algorithm to find the most appropriate set of latent parameters given the…”
Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods (2019, background): “We also note that a number of inverse graphics models [85], [86], [87] have been proposed to train disentangled representations.”
Deep Generative Vision as Approximate Bayesian Computation (2014, background): “Probabilistic formulations of inverse graphics have recently been proposed for a variety of 2D and 3D vision problems [15, 12, 14, 9].”