The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

By Jamie Watson, Oisin Mac Aodha, Victor Adrian Prisacariu, Gabriel J. Brostow, Michael Firman. Computer Vision and Pattern Recognition, 2021

370 citations recorded by the source corpus.

Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the- shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available.We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/fdf8fa52c889d70a1a43b1cf241f24736584f948

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Unifying Flow, Stereo and Depth Estimation (2022, background): “Compared with monocular methods, multi-view depth estimation can better leverage the information from additional viewpoints and usually lead to improved performance [78].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, neutral): “Self-supervised methods (Watson et al., 2021; Sun et al., 2023) are also explored to enhance temporal coherence without explicit annotations.”