Joint 3D Object and Layout Inference from a Single RGB-D Image

By Andreas Geiger, Chaohui Wang. 2015

56 citations recorded by the source corpus.

Inferring 3D objects and the layout of indoor scenes from a single RGB-D image captured with a Kinect camera is a challenging task. Towards this goal, we propose a high-order graphical model and jointly reason about the layout, objects and superpixels in the image. In contrast to existing holistic approaches, our model leverages detailed 3D geometry using inverse graphics and explicitly enforces occlusion and visibility constraints for respecting scene properties and projective geometry. We cast the task as MAP inference in a factor graph and solve it efficiently using message passing. We evaluate our method with respect to several baselines on the challenging NYUv2 indoor dataset using 21 object categories. Our experiments demonstrate that the proposed method is able to infer scenes with a large degree of clutter and occlusions.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For scenes, techniques include plane fitting [39], 3D model fitting and retrieval [15, 40] or leverage symmetries [28] and predict 3D semantics [4, 14, 59].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “This has led to the creation of large databases of real-world 3D content [7, 10, 42, 43], enabling progress in a variety of areas including 3D modeling [28, 54], 3D reconstruction [6, 52], 3D recognition [18, 44] and 3D scene understanding [16, 42].”
Semantic Scene Completion from a Single Depth Image (2016, methodology): “Moreover, since our method does not require the model fitting step it is much faster at 7s compared to 127s per image [4].”
Three-Dimensional Object Detection and Layout Prediction Using Clouds of Oriented Gradients (2016, methodology): “Previous work has used fairly elaborate, manually engineered heuristics to prune false detections [36] or used CAD models and layout cues jointly to model scenes [9].”
FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation (2017, background): “Applications of learning on point clouds include shape completion and recognition [57], unmanned autonomous vehicles [36], 3D object detection, recognition and classification [9, 33, 40, 41, 48, 49, 53], contour detection [21], layout inference [18], scene labeling [31], category discovery [60], poi”
Anisotropic Convolutional Networks for 3D Semantic Scene Completion (2020, unknown): “[7] 65.”
Superpixels: An evaluation of the state-of-the-art (2016, methodology): “We found that UE tends to correlate strongly with ASA which can be explained by Equations (3) and (6), respectively.”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, background): “Early approaches addressed the task of scene completion either without predicting semantics [16], thereby not providing a holistic understanding of the scene, or by trying to fit a fixed number of mesh models to the scene geometry [20], which limits the expressiveness of the approach.”
FoldingNet: Interpretable Unsupervised Learning on 3D Point Clouds (2017, background): “Applications of learning on point clouds include shape completion and recognition [54], unmanned autonomous vehicles [34], 3D object detection, recognition and classification [9, 31, 38, 39, 45, 46, 50], contour detection [20], layout inference [16], scene labeling [29], category discovery [57], poi”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “, for reasoning in 3D [35], [114] or to enrich 2D annotations with approximate 3D geometry.”
Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer (2015, methodology): “And finally, our 3D annotations might be useful by themselves for reasoning in 3D [14, 50] or to enrich 2D annotations with approximate 3D geometry.”
Structured Prediction of Unobserved Voxels from a Single Depth Image (2016, methodology): “are rendered from the same viewpoint, and for [39] and [17] we do not show the predicted walls and floor.”