ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

By Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Muller. arXiv.org, 2023

973 citations recorded by the source corpus.

This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains. The code and pre-trained models are publicly available at https://github.com/isl-org/ZoeDepth .

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/ebe2a5a186fa08e4e1b5d225d7a253d444d2c396

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “However, due to its limited range in depth sensing, we employ the ZoeDepth depth estimation model [33] to generate a depth image and approximate a waypoint for the detected target object.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, methodology): “We follow ZoeDepth [4] to conduct zero-shot metric depth estimation.”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “Specifically, we use ZoeDepth [4] to estimate depth map D and obtain pixel’s 3D position p = { x, y, z } in the egocentric 3D coordinate system via back-projection π − 1 with camera intrinsic parameters.”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “We perform depth estimation [6] to lift monocular 2D pixels to metric-scale 3D point clouds.”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “Given that over 95% of the video datasets for embodied tasks do not provide 3D information, we employ ZoeDepth (Bhat et al., 2023) on each frame of the video from these datasets.”
Depth Anything V2 (2024, neutral): “Notably, even our most lightweight model which is based on ViT-S, is superior to other models built on ViT-L [6].”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “The lower rows report on quantitative QAs, detailing their success rate (answers within ± 25 % of the ground truth value) and the absolute relative error [40, 39].”
Continuous 3D Perception Model with Persistent State (2025, neutral): “Extensive research [7, 27, 42, 55, 58, 59, 85, 92] focuses on estimating monocular depth.”
Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, methodology): “For example, training a ConvNeXt-XL requires 47h (160K iters) while our VPD trained for only 26h (80K iters) still performs better.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “Our baselines include Depth Anything (Yang et al., 2024a), Metric3D (Yin et al., 2023), PatchFusion (Li et al., 2024a), UniDepth (Piccinelli et al., 2024), ZeroDepth (Guizilini et al., 2023) and ZoeDepth (Bhat et al., 2023).”
Step1X-Edit: A Practical Framework for General Image Editing (2025, positive): “Color Alteration & Material Modification : After detecting objects in the image, we employ Zeodepth [4] for depth estimation to understand object geometry.”
UniDepth: Universal Monocular Metric Depth Estimation (2024, background): “Recent efforts focus on developing MMDE models [3, 20, 48] for general depth prediction across diverse domains.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “We present visualization results of our predictions (‘Ours Depth’/‘Ours Normal’), groundtruth labels (‘GT Depth’/‘GT Normal’) and results from other metric depth (‘ZoeDepth’[36]) and surface normal methods (‘Bae et al.’”