GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation

By Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, Jiaya Jia. 2018

409 citations recorded by the source corpus.

In this paper, we propose Geometric Neural Network (GeoNet) to jointly predict depth and surface normal maps from a single image. Building on top of two-stream CNNs, our GeoNet incorporates geometric relation between depth and surface normal via the new depth-to-normal and normal-to-depth networks. Depth-to-normal network exploits the least square solution of surface normal from depth and improves its quality with a residual module. Normal-to-depth network, contrarily, refines the depth map based on the constraints from the surface normal through a kernel regression module, which has no parameter to learn. These two networks enforce the underlying model to efficiently predict depth and surface normal for high consistency and corresponding accuracy. Our experiments on NYU v2 dataset verify that our GeoNet is able to predict geometrically consistent depth and normal maps. It achieves top performance on surface normal estimation and is on par with state-of-the-art depth estimation methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

IRS: A Large Synthetic Indoor Robotics Stereo Dataset for Disparity and Surface Normal Estimation (2019, background): “Recent advances [1], [2], [3], [4], [5], [6] in these vision tasks have shown that deep neural network (DNN) can significantly improve the estimation quality.”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “5D representations, such as depth map [26], image collections [19], displacement map [16] or normal map [37, 46].”
Zero-Shot Task Transfer (2019, methodology): “Surface Normal Estimation: We evaluated our method based on the evaluation criteria described in [31], [5].”
Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery (2020, background): “In addition, other researchers have explored the relationship between depth and other tasks, i.e., segmentation [18] and surface norm [26].”
Deep Surface Normal Estimation With Hierarchical RGB-D Fusion (2019, methodology): “Specifically, we choose Zhang’s method [33] for initial normal estimation from RGB, and calculate a rough normal from raw depth image at the same time, then merge the two normal estimations using methods in GFMM [10] and GeoNet [21] to estimate the final surface normal map.”
PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, positive): “Local Plane Assumption: Encouraged by these methods [22], [58],[59],weadopttheassumptionoflocalplanaritytoconstrain thelocalconsistencyofdepthandnormals,meaningapixel anditsneighboringpixelscanbeconsideredasanapproximate plane.”
Neural Window Fully-connected CRFs for Monocular Depth Estimation (2022, background): “The first approach directly regresses the continuous depth map from the aggregation of the information in an image [1, 6, 12, 17, 18, 26, 28, 39].”
Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries (2018, methodology): “The methods denoted without a superscript [9, 16, 40, 25, 2, 3, 24, 21, 32, 10] and ours should be able to be compared in an equal condition.”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, methodology): “Depth Estimation: Many works have been proposed for monocular depth estimation [10, 11, 37, 32, 42, 29, 63, 54, 47, 60, 58, 46, 62].”
Consistent video depth estimation (2020, methodology): “…coupled training with optical flow [Ranjan et al. 2019; Yin and Shi 2018; Zou et al. 2018], object motion [Dai et al. 2019; Vijayanarasimhan et al. 2017], surface normal [Qi et al. 2018], edge [Yang et al. 2018], and visual odometry [Andraghetti et al. 2019; Shi et al. 2019; Wang et al. 2018b].”
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations (2018, background): “[33] found it beneficial to directly encode a geometrical structure as part of the network architecture in order to perform depth estimation and surface normals estimation simultaneously.”
FastDepth: Fast Monocular Depth Estimation on Embedded Systems (2019, background): “[18] trained networks to estimate both the depth and the normals, as a way to address the problem of blurriness in predictions.”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “[39] jointly predicted depth and surface normal based on two-stream CNNs and focused on indoor scenes.”
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (2019, neutral): “[33] 0.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, background): “Some papers incorporate additional cues such as normal [135], segmentation [68], and edge [76] maps, to guide the disparity estimation.”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “Some existing approaches propose to explicitly acquire the surface normal from the depth map by using some geometric constraints, such as Sobel-like operator [19, 27], differentiable least square [36, 44], or randomly sampled point triplets [37, 69, 70].”