Learning 3-D Scene Structure from a Single Still Image

By Ashutosh Saxena, Min Sun, Andrew Y. Ng. 2007

557 citations recorded by the source corpus.

We consider the problem of estimating detailed 3D structure from a single still image of an unstructured environment. Our goal is to create 3D models which are both quantitatively accurate as well as visually pleasing. For each small homogeneous patch in the image, we use a Markov random field (MRF) to infer a set of "plane parameters" that capture both the 3D location and 3D orientation of the patch. The MRF, trained via supervised learning, models both image depth cues as well as the relationships between different parts of the image. Inference in our model is tractable, and requires only solving a convex optimization problem. Other than assuming that the environment is made up of a number of small planes, our model makes no explicit assumptions about the structure of the scene; this enables the algorithm to capture much more detailed 3D structure than does prior art (such as Saxena et ah, 2005, Delage et ah, 2005, and Hoiem et el, 2005), and also give a much richer experience in the 3D flythroughs created using image-based rendering, even for scenes with significant non-vertical structure. Using this approach, we have created qualitatively correct 3D models for 64.9% of 588 images downloaded from the Internet, as compared to Hoiem et al.'s performance of 33.1%. Further, our models are quantitatively more accurate than either Saxena et al. or Hoiem et al.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation (2019, methodology): “imation on NYUD-v2. The second column indicates amount of supervised target samples used during training. Method sup. Error# Accuracy "( = 1:25) rel log10 rms < < 2 < 3 Saxena et al. [51] 795 0.349 - 1.214 0.447 0.745 0.897 Liu et al. [33] 795 0.230 0.095 0.824 0.614 0.883 0.975 Eigen ”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “However, collecting large and varied training datasets with accurate ground truth depth for supervised learning [56, 10] is itself a formidable challenge.”
Make3D: Learning 3D Scene Structure from a Single Still Image (2009, methodology): “1Parts of this work were presented in [1], [2] and [3].”
A stochastic graph grammar for compositional object representation and recognition (2009, methodology): “Significant work has been done recently using structure-based approaches to perform general image recognition, including the tasks of scene segmentation and structure inference [5,14].”
StereoScan: Dense 3d reconstruction in real-time (2011, methodology): “With the seminal work by Hoiem et al. [14] learningbased approaches to geometry estimation from monocular images have seen a revival [23], [13].”
Learning Spatial Context: Using Stuff to Find Things (2008, methodology): “[22], allowing us to utilize 3D context and address the issue of scale.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, unknown): “Make3D [14] 2009 Real Monocular depth 2272× 1704 534 monocular 400 134 78× 51 534 Dense − − KITTI2012 [15] 2012 Real Stereo 1240× 376 389 2 194 195 1226× 370 − Sparse − − Y Y MPI Sintel [16] 2012 Synthetic Optical flow 1024× 436 35 videos 50 23 videos 12 videos − − Dense − − NYU2 [17] 2012 Real - in”
A Fast Data Collection and Augmentation Procedure for Object Recognition (2008, unknown): “We first apply our synthetic data augmentation techniques to place the object (tree) correctly in new backgrounds using 3D depth information , (Saxena, Sun, and Ng 2007) producing a much larger augmented training set.”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “tensively in the computer vision and graphics communities (Saxena et al. 2009; Chaudhuri et al. 2011; Kalogerakis et al. 2012; Chang et al. 2015; Rezende et al. 2016; Soltani et al. 2017; Kulkarni et al. 2015; Tulsiani et al. 2016; Huang et al. 2019; Jiang et al. 2019).”
Robotic Grasping of Novel Objects using Vision (2008, methodology): “In Saxena et al. (2007b), we presented an algorithm for predicting the 3-d orientation of an object from its image.”