Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

By David Eigen, Rob Fergus. 2014

2,848 citations recorded by the source corpus.

In this paper we address three different computer vision tasks using a single basic architecture: depth prediction, surface normal estimation, and semantic labeling. We use a multiscale convolutional network that is able to adapt easily to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Examples include: object detection [1,4], semantic segmentation [5,6], human pose estimation [7,8], video recognition [9], monocular depth estimation [10], and so on.”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “(2016), CNN-LSTM in Gan et al. (2016), Skipthought in Kiros et al. (2015), Word embedding average in Arora et al.”
Contrastive Multiview Coding (2019, background): “The views we consider are: luminance (L channel), chrominance (ab channel), depth, surface normal [20], and semantic labels.”
Unified Perceptual Parsing for Scene Understanding (2018, background): “Eigen and Fergus [23] propose an architecture that jointly addresses depth prediction, surface normal estimation, and semantic labeling.”
Learning to Refine Object Segments (2016, methodology): “Features can be computed independently at each scale [13], or the output from one scale can be used as additional input to the next finer scale [14,15].”
COCO-Stuff: Thing and Stuff Classes in Context (2016, methodology): “To evaluate performance we use four criteria commonly used in the literature [37, 16, 8]: (1) pixel accuracy is the percentage of correctly labeled pixels in the”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “, [14, 23, 18, 8, 10, 50, 30, 36, 16, 55, 58]), despite the fact that the setting is always the same: predict pixels from pixels.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “The segmentationfree approaches of [14], [52] directly apply DCNNs to the”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “Years of research in computer vision, image processing, computational photography, and graphics have produced powerful translation systems in the supervised setting, where example image pairs { x, y } are available (Figure 2, left), e.g., [10, 18, 21, 22, 26, 31, 44, 55, 57, 61].”
WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, methodology): “We run the semi-supervised approach from [33] using the model proposed by Eigen [14] as baseline on our much larger dataset and obtained an RMSE (Root Mean Square Error) value of 7.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Recent methods for depth [5,6,18] and surface normal estimation [6,19] are similar, transforming color input images into geometrically meaningful output images using a feed-forward convolutional network trained with per-pixel regression [5,6] or classification [19] losses.”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “…with the spatial localization issues outlined in the beginning of the introduction, Long et al. (2014) upsample and concatenate the scores from inter-mediate feature maps, while Eigen & Fergus (2014) reﬁne the prediction result from coarse to ﬁne by propagating the coarse results to another DCNN.”
Conditional Random Fields as Recurrent Neural Networks (2015, background): “They later extended this work, and developed a joint approach [5] that simultaneously pixel-wisely predict the depth, surface normal and semantic labels.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”