Learning to Recover 3D Scene Shape from a Single Image

By Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, Chunhua Shen. Computer Vision and Pattern Recognition, 2020

325 citations recorded by the source corpus.

Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length. We investigate this problem in detail, and propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then use 3D point cloud encoders to predict the missing depth shift and focal length that allow us to recover a realistic 3D scene shape. In addition, we propose an image-level normalized regression loss and a normal-based geometry loss to enhance depth prediction models trained on mixed datasets. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot dataset generalization. Code is available at: https://git.io/Depth

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/4ac318309807b5a7a50a1a292ced43e6f5f0ffbb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “Since monocular depth estimation is an inherently ill-posed task due to scale and shift ambiguity, we standardize the depth values in a robust way by ignoring the top and bottom 10% of values [78].”
Automated distance estimation for wildlife camera trapping (2022, methodology): “Contrary to (Yin et al., 2021), we presume that the focal length of the camera is known, since the focal length of commercial camera traps can usually be found in the specifications provided by the manufacturer or can be easily calculated from the opening angle of the camera (Field Of View, FOV)…”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “These methods systematically build on or re-use existing monocular depth estimation (MDE) networks [6, 91, 169, 171].”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “These methods are specialized in predicting either depth (Di-verseDepth [71], LeReS [73], HDN [76], Marigold [24]) or disparity (MiDaS [48], DPT [47], Omnidata [11]).”
UniDepth: Universal Monocular Metric Depth Estimation (2024, methodology): “On the other hand, scale-agnostic depth methods, including MiDaS [35], Om-niData [12], and LeReS [47], show robust generalization by training on extensive datasets.”
Depth Anything V2 (2024, unknown): “Other works [94, 96, 89] focus on the data-driven perspective.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, unknown): “We propose to randomly crop several patches from images and enforce the scale-shift invariant loss [25], [27] on them.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “LeReS (Yin et al., 2021) trains a separate network for undistorting point clouds to recover scale and shift, Metric3D (Yin et al., 2023) scales images or depth maps to a canonical space and remaps estimated depth given the focal length, and ZeroDepth (Guizilini et al., 2023) learns camera-specific…”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “Di-verseDepth [56], LeReS [57] and HDN [60] estimate affine-invariant depth maps, while MiDaS [35], DPT [36], and Om-nidata [13] produce affine-invariant disparities.”
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, unknown): “[49] recover 3D scene shape from a single image via a two-stage framework combining monocular depth estimation with 3D point cloud encoders that are trained to predict missing depth shift and focal length.”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, methodology): “We propose to randomly crop several patches from images and enforce the scale-shift invariant loss [69, 40] on them.”
MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2024, unknown): “For a comprehensive comparison with different monocular depth estimation methods, we evaluate scale-invariant depth [4, 24, 38, 52, 64], affine-invariant depth [18, 27], and affine-invariant disparity [5, 60, 61].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Early approaches to video depth estimation (Luo et al., 2020; Zhang et al., 2021) improve temporal consistency by fine-tuning depth models, and sometimes motion models, at test time for each input video.”