Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

By Shuran Song, Andy Zeng, Angel X. Chang, Manolis Savva, Silvio Savarese, Thomas A. Funkhouser. 2017

79 citations recorded by the source corpus.

We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360° panoramic view of an indoor scene when given only a partial observation (= 50%) in the form of an RGB-D image. To make this possible, Im2Pano3D leverages strong contextual priors learned from large-scale synthetic and real-world indoor scenes. To ease the prediction of 3D structure, we propose to parameterize 3D surfaces with their plane equations and train the model to predict these parameters directly. To provide meaningful training supervision, we use multiple loss functions that consider both pixel level accuracy and global context consistency. Experiments demonstrate that Im2Pano3D is able to predict the semantics and 3D structure of the unobserved scene with more than 56% pixel accuracy and less than 0.52m average distance error, which is significantly better than alternative approaches.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Geology

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “mas and CAD models [69, 70, 71, 72], which are usually curated datasets of human-taken photos [53].”
Spherical CNNs on Unstructured Grids (2019, background): “Song et al. (2017) study semantic segmentation of equirectangular omni-directional images, but in the context of image inpainting, where only a partial view is given as input.”
Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery (2020, background): “Im2Pano3d [28] generates a 360 room with a partial perspective observation as input.”
ShapeCodes: Self-supervised Feature Learning by Lifting Views to Viewgrids (2017, unknown): “, via disentanglement with deep inverse graphic networks [48], tensor completion [49], recurrent encoder-decoder nets [50,51], appearance flow [18], or converting partial RGBD to panoramas [52].”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “in navigation [29, 2]) would be interesting future work.”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, background): “More recent works study how to obtain 3D data from color images by directly inferring depth images [14, 29, 10, 40, 22, 53, 16, 45], or filling in missing depth values in RGB-D images captured by commodity 3D cameras [21, 17, 63].”
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks (2017, background): “However, whereas [57] considers completion tasks where about half of the scene is observed, in our case only about 5% of the scene is observed at a time.”
Sidekick Policy Learning for Active Visual Exploration (2018, background): “Methods for depth estimation [22,16,60] and/or semantic segmentation [56,25,26] use RGBD depth data, multiple views, and/or auxiliary annotations during training, then proceed with single image observations at test time.”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, methodology): “This is representative of scene completion approaches [68,78].”