Marr Revisited: 2D-3D Alignment via Surface Normal Prediction

By Aayush Bansal, Bryan C. Russell, Abhinav Kumar Gupta. 2016

235 citations recorded by the source corpus.

We introduce an approach that leverages surface normal predictions, along with appearance cues, to retrieve 3D models for objects depicted in 2D still images from a large CAD object library. Critical to the success of our approach is the ability to recover accurate surface normals for objects in the depicted scene. We introduce a skip-network model built on the pre-trained Oxford VGG convolutional neural network (CNN) for surface normal prediction. Our model achieves state-of-the-art accuracy on the NYUv2 RGB-D dataset for surface normal prediction, and recovers fine object detail compared to previous methods. Furthermore, we develop a two-stream network over the input image and predicted surface normals that jointly learns pose and style for CAD model retrieval. When using the predicted surface normals, our two-stream network matches prior work using surface normals computed from RGB-D images on the task of pose prediction, and achieves state of the art when using RGB-D input. Finally, our two-stream network allows us to retrieve CAD models that better match the style and pose of a depicted object compared with baseline approaches.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “Setup: We use the surface normal estimation task [22], with the evaluation, and dataset splits as formulated in [5, 45, 68].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “Competing methods, e.g. [2, 25] are using alternative normal estimation methods for the extended data, but we would not expect the differences because of this to be too large.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…[McCormac et al., 2017, Silberman et al., 2012, Song et al., 2017], many papers start to estimate depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Janner et al., 2017, Shi et al., 2017] with deep networks.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “With the development of depth sensors [23] and larger-scale RGB-D datasets [33,42,43], there have also been papers on estimating depth [6,10], surface normals [1,54], and other intrinsic images [25,41] with deep networks.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Non-parametric approaches retrieve shapes from a large corpus [2, 20, 23], but require having an exact instance in the corpus.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “…et al., 2012, Song et al., 2017, McCormac et al., 2017], there have also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Shi et al., 2017, Janner et al., 2017] with deep networks.”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “…can be used for 3D shape classiﬁcation [Su et al., 2015a, Sharma et al., 2016, Maturana and Scherer, 2015], 3D shape retrieval [Shi et al., 2015, Su et al., 2015a], and single image 3D reconstruction [Kar et al., 2015, Bansal et al., 2016, Girdhar et al., 2016], mostly with full supervision.”
Single Image 3D Interpreter Network (2016, background): “Depth-/mesh-based models can recover detailed 3D object structure from a single image, either by adapting existing 3D models from a database [3,19,6,8,20,21,22,23,24], or by inferring from its detected 2D silhouette [5,7,4].”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “Networks have been trained to perform that task us-ing RGB-D datasets in the past [9, 23, 3, 41, 48].”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “We directly pretrain on our synthetic data, followed by ﬁnetuning on NYUv2 similar to the [5].”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, background): “While predicted surface normals are useful for tasks like shading [23], 2D-3D alignment [3], and face morphing [28], it alone is insufﬁcient to describe an object’s complete 3D geometry, making it difﬁcult to be directly used by manipulation algorithms that require 3D data ( e . g . depth images,…”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, unknown): “Given a scene, one can predict interactions using geometrybased rules for objects via 3D scene understanding [43, 78, 79, 134], estimating 3D physical attributes [8, 26, 41, 137] or through segmentation models trained on semantic interactions [102, 104], and thus require specialized datasets.”
Zero-Shot Task Transfer (2019, methodology): “SkipNet [5] 20.”
GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation (2018, background): “The evaluation metrics for surface normal prediction [33, 3, 7] are mean of angle error (mean), medians of the angle error (median), root mean square error (rmse), and pixel accuracy as percentage of pixels with angle error below threshold t where t ∈ [11.”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, background): “(a) image; (b) predictions of [10]; (c) predictions of [1] ; (d) predictions of [46]; (e) our results; (f) ground truth.”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, background): “5D representations, such as depth map [8, 39, 42, 7, 24] or normal map [1, 50, 34, 51].”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, methodology): “(ii) Align object proposals to RGB or depth image by treating objects as geometric primitives or CAD models [3, 39, 57].”
Learning to predict indoor illumination from a single image (2017, methodology): “CNN-based methods have also been proposed for estimating scene depth [Eigen and Fergus 2015], surface normals [Bansal et al. 2016; Eigen and Fergus 2015], and intrinsic decompositions for indoor scenes [Zhou et al. 2015].”