Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image

By Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, Chunhua Shen. IEEE International Conference on Computer Vision, 2023

446 citations recorded by the source corpus.

Reconstructing accurate 3D scenes from images is a long-standing vision task. Due to the ill-posedness of the single-image reconstruction problem, most well-established methods are built upon multi-view geometry. State-of-the-art (SOTA) monocular metric depth estimation methods can only handle a single camera model and are unable to perform mixed-data training due to metric ambiguity. Meanwhile, SOTA monocular methods trained on large mixed datasets achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. In this work, we show that the key to a zero-shot single-view metric depth model lies in the combination of large-scale data training and resolving the metric ambiguity from various camera models. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problems and can be effortlessly plugged into existing monocular models. Equipped with our module, monocular models can be stably trained over 8 millions of images with thousands of camera models, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Experiments demonstrate SOTA performance of our method on 7 zero-shot benchmarks. Notably, our method won the championship in the 2nd Monocular Depth Estimation Challenge. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. The potential benefits extend to downstream tasks, which can be significantly improved by simply plugging in our model. For example, our model relieves the scale drift issues of monocular-SLAM (Fig. 1), leading to high-quality metric scale dense mapping. The code is available at https://github.com/YvanYin/Metric3D.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/bdd2972730730844d0366a5e5f596b1aeaa7c3ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “Building upon prior research [72], we assess the performance of depth estimation methods using the absolute relative error (AbsRel) and accuracy within a threshold δ 1 = 1 .”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “Another way is to explicitly learn to predict camera intrin-sics, which enables to perform metric 3D reconstruction from a single image when combined with MDE [168, 170].”
Depth Anything V2 (2024, unknown): “For example, MiDaS [56, 55, 7] and Metric3D [95] collect 2M and 8M labeled images respectively.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “Our baselines include Depth Anything (Yang et al., 2024a), Metric3D (Yin et al., 2023), PatchFusion (Li et al., 2024a), UniDepth (Piccinelli et al., 2024), ZeroDepth (Guizilini et al., 2023) and ZoeDepth (Bhat et al., 2023).”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “Recently, some works [4, 22, 78] take a step further to estimate the metric depth.”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “Importantly, visual cues for depth depend not only on the scene content but also on the (generally unknown) camera intrinsics [58].”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, unknown): “Modern generalist approaches [6, 42, 95, 112, 113, 118] exemplify this trend by leveraging massive multi-dataset training and advanced architectures like vision transformers [67] or DiT [64].”
UniDepth: Universal Monocular Metric Depth Estimation (2024, methodology): “These models often leverage camera awareness, either by directly incorporating external camera parameters into computations [14, 20] or by normalizing the shape or output depth based on intrinsic properties, as seen in [1, 24, 48].”