DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding

By Yinda Zhang, Mingru Bai, Pushmeet Kohli, Shahram Izadi, Jianxiong Xiao. 2016

78 citations recorded by the source corpus.

3D context has been shown to be extremely important for scene understanding, yet very little research has been done on integrating context information with deep neural network architectures. This paper presents an approach to embed 3D context into the topology of a neural network trained to perform holistic scene understanding. Given a depth image depicting a 3D scene, our network aligns the observed scene with a predefined 3D scene template, and then reasons about the existence and location of each object within the scene template. In doing so, our model recognizes multiple objects in a single forward pass of a 3D convolutional neural network, capturing both global scene and local object information simultaneously. To create training data for this 3D network, we generate partially synthetic depth images which are rendered by replacing real objects with a repository of CAD models of the same object category1. Extensive experiments demonstrate the effectiveness of our algorithm compared to the state of the art.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

OctNet: Learning Deep 3D Representations at High Resolutions (2016, methodology): “Similar models have also been exploited for semantic point cloud labeling [20] and scene context has been integrated in [51].”
The limits and potentials of deep learning for robotics (2018, methodology): “More recent work [136] devises a method to perform holistic scene understanding using a deep neural network that learns to utilize context information from training data.”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “This has recently led to utilizing synthetic data in the form of 2D render pairs (RGB image and per-pixel label map) from digital 3D models [4, 8, 13, 31, 25, 19].”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “Synthetic data has been successfully used in a range of application domains including prediction of object pose [33, 21, 11], optical flow [6], semantic segmentation for indoor scenes [12, 36], and analysis of image features [2, 15].”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, background): “Synthetic image datasets have recently been a source of training data for object detection and correspondence matching [26,32,37,83,91,95,113,117,120,150], singleview reconstruction [58], view-point estimation [84, 119], 2D human pose estimation [93,96,105], 3D human pose estimation [18, 27, 39, 104”
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation (2019, background): “Category-Level 3D Object Detection: One of the challenges in predicting the 6D pose and size of objects is localizing them in the scene and finding their physical sizes, which can be formulated as a 3D detection problem [52, 21, 20, 30, 13].”