Designing deep networks for surface normal estimation

By X. Wang, David F. Fouhey, Abhinav Kumar Gupta. 2014

378 citations recorded by the source corpus.

In the past few years, convolutional neural nets (CNN) have shown incredible promise for learning visual representations. In this paper, we use CNNs for the task of predicting surface normals from a single image. But what is the right architecture? We propose to build upon the decades of hard work in 3D scene understanding to design a new CNN architecture for the task of surface normal estimation. We show that incorporating several constraints (man-made, Manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation. We also show that our network is quite robust and show state of the art results on other datasets as well without any fine-tuning.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “After the resurgence of ConvNets for image classification [8, 29], they have been successfully adopted for a variety of computer vision tasks such as object detection [16, 17, 41, 48], semantic segmentation [3, 6, 34, 35], instance segmentation [21, 22, 38], pose estimation [49, 50], depth estimatio”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “Setup: We use the surface normal estimation task [22], with the evaluation, and dataset splits as formulated in [5, 45, 68].”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “For example, networks trained on RGB images have been shown to perform well in tasks such as image segmentation [40], semantic segmentation [24], image colorization [42, 23, 16], depth prediction [7], surface normal prediction [7, 37], and image stylization [19].”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Recent methods for depth [5,6,18] and surface normal estimation [6,19] are similar, transforming color input images into geometrically meaningful output images using a feed-forward convolutional network trained with per-pixel regression [5,6] or classification [19] losses.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, methodology): “ set of normals and mixture coefﬁcients. Barron and Malik [3,2] infer normals from RGB-D inputs using a set of handcrafted priors, along with illumination and reﬂectance. From RGB inputs, Wang et al. [38] use convolutional networks to combine normals estimates from local and global scales, while als”
Cross-Stitch Networks for Multi-task Learning (2016, methodology): “pose estimation and action recognition [22], surface normals and edge labels [52], face landmark detection and face detection [57, 59], auxiliary tasks in detection [21], related”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “Even though our multi-task network’s performance is not too different from the plain CNN-based result of [103], it is clear that there we have a somewhat unique gap in performance, when compared to what we seen in the remaining tasks.”
Unsupervised Learning of Visual Representations Using Videos (2015, methodology): “Note that unlike [52], we do not utilize any data from the videos in NYU dataset for training.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “More specifically, we quantize the surface normals to 40 classes with k-means clustering as in [39,54] and the loss is defined as”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…[McCormac et al., 2017, Silberman et al., 2012, Song et al., 2017], many papers start to estimate depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Janner et al., 2017, Shi et al., 2017] with deep networks.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “With the development of depth sensors [23] and larger-scale RGB-D datasets [33,42,43], there have also been papers on estimating depth [6,10], surface normals [1,54], and other intrinsic images [25,41] with deep networks.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “…et al., 2012, Song et al., 2017, McCormac et al., 2017], there have also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Shi et al., 2017, Janner et al., 2017] with deep networks.”
Learning a Predictable and Generative Vector Representation for Objects (2016, methodology): “Note that our method automatically predicts occluded regions of the object, unlike most work on single image 3D (e.g., [14,13,29,30,31,32]) that predict a 2.5D shell.”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, methodology): “For the loss function, we follow the design in [55].”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, methodology): “A similar observation has been made by [38,37] that formulate a continuous variable estimation problem as classification.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “Networks have been trained to perform that task us-ing RGB-D datasets in the past [9, 23, 3, 41, 48].”
The Curious Robot: Learning Visual Representations via Physical Interactions (2016, background): “Finally, our approach is also related to other efforts which train deep networks using supervision from other sensors such as Kinect [13,14] or motion information such as optical flow [15].”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, methodology): “We performed left-right flipping of the image and color augmentation, similar to [46], over the NYU trainval frames only; we did not perform augmentation over the video frames.”
Deep Stereo: Learning to Predict New Views from the World's Imagery (2015, methodology): “hich uses images with manually annotated geometric classes. More recent methods have used Kinect data for training [15,18] and deep learning methods for single view depth or surface normal prediction [9,33]. However, the single-view problem remains very challenging. Moreover, gathering sufﬁcient tra”