Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

By Shubham Tulsiani, Saurabh Gupta, David F. Fouhey, Alexei A. Efros, Jitendra Malik. 2017

139 citations recorded by the source corpus.

The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose. We propose a convolutional neural network-based approach to predict this representation and benchmark it on a large dataset of indoor scenes. Our experiments evaluate a number of practical design questions, demonstrate that we can infer this representation, and quantitatively and qualitatively demonstrate its merits compared to alternate representations.

Subfields: Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “Tulsiani et al. [138], on the other hand, proposed an approach that is entirely based on deep learning.”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, unknown): “Most works in this area focus on separately re-constructing the human mesh [1, 10, 11, 15, 16, 27] or the individual 3D object [4, 18, 22, 31].”
SynSin: End-to-End View Synthesis From a Single Image (2019, background): “A second vein of work assumes a large dataset of images with corresponding ground-truth 3D and semantic information to train their 3D representation [40, 52, 62].”
Learning Physical Graph Representations from Visual Scenes (2020, methodology): “3D-RelNet [29] further extended Factor3D to model object relations.”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Some approaches [50, 21, 11, 22] utilize neural network to predict the object shape, translation, and global orientation in the camera coordinate.”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, background): “While our approach predicts floors, this is learned, not explicitly baked in, unlike modern inheritors that have explicit object and layout components [27, 50] or the ability to query for two layers [25].”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “There are several works [60,39,16] that recover the 3D shape of multiple objects but still reason about them independently.”