Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

By René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun. 2019

2,643 citations recorded by the source corpus.

The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible. In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks. Armed with these tools, we experiment with five diverse training datasets, including a new, massive data source: 3D films. To demonstrate the generalization power of our approach we use zero-shot cross-dataset transfer, i.e. we evaluate on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources greatly improves monocular depth estimation. Our approach clearly outperforms competing methods across diverse datasets, setting a new state of the art for monocular depth estimation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Projected GANs Converge Faster (2021, neutral): “Using pretrained representations has become ubiquitous in computer vision [29, 30, 48]”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “This has many broader applications, such as navigation and visual odometry in robots [11], estimation of 3D geometry [66], and even to aid transfer of more complex, learned inference such as 3D human pose estimation from synthetic to real images [23].”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “1 reports results for unconditional image modeling on ImageNet (IN) [14], Restricted ImageNet (RIN) [57], consisting of a subset of animal classes from ImageNet, LSUN Churches and Towers (LSUN-CT) [69], and for conditional image modeling of RIN conditioned on depth maps obtained with the approach of”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “Remove Objects from Image [10, 35] Replace Objects from Image [10, 35] Change Image by the Text [35] Image Question Answering [23] Image-to-Text [23] Text-to-Image [35] Image-to-Edge [45] Edge-to-Image [53] Image-to-Line [16] Line-to-Image [53] Image-to-Hed [44] Hed-to-Image [53] Image-to-Seg [24] S”
Vision Transformers for Dense Prediction (2021, methodology): “We construct a meta-dataset that includes the original datasets that were used in [32] (referred to as MIX 5 in that work) and extend it with with five additional datasets ([19, 45, 46, 48, 49]).”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, methodology): “[16] use multiple datasets and mix them via a multi-task learning framework.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “On Hypersim, we use a MiDaS [51] DPT-Hybrid, since the Omnidata depth model was partially trained on this dataset.”
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, methodology): “We compare models with and without pre-training for relative depth as in [33].”
GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “For Level-1, we use Tag2Text [14] and RAM [58] for image tagging, Co-DETR [62], EVAv02 [7], OWL-ViT [33], and POMP [40] for object localization, GRiT [48] and GPT4RoI [57] for attribute generation, and MiDAS [39] for depth estimation.”
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, positive): “Normalized 3D pointwise regression loss: The normalized regression loss for X is a multi-view version of that in DUST3R [66] or monocular depth estimation [15, 36, 66].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Recent deep learning works (Ranftl et al., 2020; 2021; Saxena et al., 2024; Ke et al., 2024) target zero-shot performance and with large-scale training combined with synthetic datasets (Yang et al., 2024a;b) show strong generalization to diverse domains.”
Depth Anything V2 (2024, positive): “Building on the pioneering work of MiDaS [56, 7] in zero-shot MDE, recent studies tend to construct larger-scale training datasets in an effort to enhance estimation performance.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “To perform the task, we estimate optical flow from RAFT [58] and produce monocular depth maps from MI-DAS [49], and then normalize and concatenate on the channel dimension.”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, neutral): “2 [48] René Ranftl, Katrin Lasinger, David Hafner, Konrad”
Continuous 3D Perception Model with Persistent State (2025, neutral): “Extensive research [7, 27, 42, 55, 58, 59, 85, 92] focuses on estimating monocular depth.”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, background): “In monocular depth estimation, state-of-the-art models rely on sophisticated scale-invariant depth losses [11, 13, 33, 34].”