Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras

By Ariel Gordon, Hanhan Li, Rico Jonschkowski, Anelia Angelova. 2019

399 citations recorded by the source corpus.

We present a novel method for simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos, using only consistency across neighboring video frames as supervision signal. Similarly to prior work, our method learns by applying differentiable warping to frames and comparing the result to adjacent ones, but it provides several improvements: We address occlusions geometrically and differentiably, directly using the depth maps as predicted during training. We introduce randomized layer normalization, a novel powerful regularizer, and we account for object motion relative to the scene. To the best of our knowledge, our work is the first to learn the camera intrinsic parameters, including lens distortion, from video in an unsupervised manner, thereby allowing us to extract accurate depth and motion from arbitrary videos of unknown origin at scale. We evaluate our results on the Cityscapes, KITTI and EuRoC datasets, establishing new state of the art on depth prediction and odometry, and demonstrate qualitatively that depth prediction can be learned from a collection of YouTube videos. The code will be open sourced once anonymity is lifted.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Recent work [8] that explicitly reasons about camera / ego motion could be integrated to globally transport features between source and target frames.”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “[41] estimate the intrinsic parameters of YouTube videos in Fig.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “We also do not make use of any trained semantic models to mask moving objects e.g. [5, 26, 29].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Self-supervised approaches (Gordon et al., 2019; Mahjourian et al., 2018; Go-dard et al., 2019; Yang et al., 2018) learn these tasks through self-supervised proxy tasks.”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “traditional matching and optimization-based methods, current research on this topic mainly focuses on deep learning in an unsupervised or self-supervised way [132].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “For general image data it was proposed to additionally learn camera calibration parameters [22] or utilize additional depth labels from synthetic data [4].”
Monocular depth estimation based on deep learning: An overview (2020, background): “ted images as input. Since the above methods are based on the prerequisites of known camera intrinsic parameters, this limits the application of the network to unknown cameras. Therefore, in [91] and [92], they extend the pose network to estimate the camera intrinsic parameter and further reduce the”
Consistent video depth estimation (2020, background): “Other notable efforts include using stereo information [Guo et al. 2018; Watson et al. 2019], better network architecture and training loss design [Gordon et al. 2019; Guizilini et al. 2019], scale-consistent ego-motion network [Bian et al. 2019], incorporating 3D geometric constraints [Mahjourian…”