Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation

By Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, Shaojie Shen. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

489 citations recorded by the source corpus.

We introduce Metric3D v2, a geometric foundation model designed for zero-shot metric depth and surface normal estimation from single images, critical for accurate 3D recovery. Depth and normal estimation, though complementary, present distinct challenges. State-of-the-art monocular depth methods achieve zero-shot generalization through affine-invariant depths, but fail to recover real-world metric scale. Conversely, current normal estimation techniques struggle with zero-shot performance due to insufficient labeled data. We propose targeted solutions for both metric depth and normal estimation. For metric depth, we present a canonical camera space transformation module that resolves metric ambiguity across various camera models and large-scale datasets, which can be easily integrated into existing monocular models. For surface normal estimation, we introduce a joint depth-normal optimization module that leverages diverse data from metric depth, allowing normal estimators to improve beyond traditional labels. Our model, trained on over 16 million images from thousands of camera models with varied annotations, excels in zero-shot generalization to new camera settings. As shown in Fig. 1, It ranks the 1st in multiple zero-shot and standard benchmarks for metric depth and surface normal prediction. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. Our model also relieves the scale drift issues of monocular-SLAM (Fig. 3), leading to high-quality metric scale dense mapping. Such applications highlight the versatility of Metric3D v2 models as geometric foundation models.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/ee531657945a58a1b4b456a6834917c0bc4025e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Hongxu Yin
2. Ruihan Yang
3. Yang Fu
4. Jan Kautz
5. Xiaolong Wang
6. Sifei Liu
7. Qiushan Guo
8. An-Chieh Cheng

In Media Technology

1. Hongxu Yin
2. Jan Kautz

Citations in the corpus

• Continuous 3D Perception Model with Persistent State (2025, unknown): “While depth maps provide valuable 3D information, converting them into 3D point clouds typically requires camera intrinsics [8, 32, 55], with camera extrinsics often left unmodeled.”
• Depth Anything V2 (2024, unknown): “One group [7, 6, 89, 28] is based on discriminative models, e . g ., BEiT [4] and DINOv2 [50], while the other [31, 20, 25] is based on generative models, e . g ., Stable Diffusion (SD) [59].”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “Recently, Metric3Dv2 [42] takes focal length as input and is trained end-to-end to predict metric depth and surface normals.”
• Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “Input image on top, estimated depth maps from Depth Pro, Marigold (Ke et al., 2024), Depth Anything v2 (Yang et al., 2024b), and Metric3D v2 (Hu et al., 2024) below.”