DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency

By Yuliang Zou, Zelun Luo, Jia-Bin Huang. European Conference on Computer Vision, 2018

504 citations recorded by the source corpus.

We present an unsupervised learning framework for simultaneously training single-view depth prediction and optical flow estimation models using unlabeled video sequences. Existing unsupervised methods often exploit brightness constancy and spatial smoothness priors to train depth or flow models. In this paper, we propose to leverage geometric consistency as additional supervisory signals. Our core idea is that for rigid regions we can use the predicted scene depth and camera motion to synthesize 2D optical flow by backprojecting the induced 3D scene flow. The discrepancy between the rigid flow (from depth prediction and camera motion) and the estimated flow (from optical flow model) allows us to impose a cross-task consistency loss. While all the networks are jointly optimized during training, they can be applied independently at test time. Extensive experiments demonstrate that our depth and flow models compare favorably with state-of-the-art unsupervised methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/c01876292b5d1ce6e746fd2e2053453847905bb2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, methodology): “Geonet [37] and DF-Net [39] are more similar to our architecture but use a larger ResNet-50 architecture.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Other works, instead, directly tackle highly complex scenarios [67] and model rigid and non-rigid components present in the scene using the estimated depth, relative camera poses, and optical ﬂow in order to handle independent motions [47, 64, 72, 5, 39, 61] or by means of scene decomposition [41].”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, methodology): “estimation have mostly focused on engineering the loss function [5, 33, 47, 53], we show that performance critically depends on the model architecture, in line with the observations of [27] for other self-supervised tasks.”
Digging Into Self-Supervised Monocular Depth Estimation (2018, background): “in the form of known object sizes [66], sparse ordinal depths [77, 6], supervised appearance matching terms [72, 73], or unpaired synthetic depth data [45, 2, 16, 78], all while still requiring the collection of additional depth or other annotations.”
Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation (2019, unknown): “[45] N Y 0.”
Unifying Flow, Stereo and Depth Estimation (2022, result): “We also note that our unified model is different from multi-task joint training approaches [81], [82], [83], [84], [85].”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Seminal work in this direction is [82], extended by leveraging on point-cloud alignment [45], differentiable DVO [69], optical flow [78, 83, 11, 3], semantic [66] or scale consistency [5].”
Consistent video depth estimation (2020, methodology): “Recent methods improve the performance through incorporating coupled training with optical flow [Ranjan et al. 2019; Yin and Shi 2018; Zou et al. 2018], object motion [Dai et al.”
Self-Supervised Monocular Depth Hints (2019, background): “Depth regression is optimized and relative poses come from stereo camera calibration in a training-from-stereo setting [9, 7, 25, 27, 26], while depth values and camera poses can be optimized jointly when training on videos [42, 20, 37, 22, 38, 32, 44, 36, 28].”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, background): “depth and flow estimation [80], detection and classification [81], [82], detection and segmentation [2], [83].”
Attentive Single-Tasking of Multiple Tasks (2019, background): “Joint learning is beneﬁcial for unsupervised learning [45], when tasks provide complementary information (eg. depth boundaries and motion boundaries [70]), in cases where task A acts as regularizer for task B due to limited data [32], or in order to learn more generic representations from synthetic…”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “In motion analysis, forward-backward consistency check can be used for detecting occlusion [28, 20, 52] or learning visual correspondence [45].”
Depth-Aware Video Frame Interpolation (2019, background): “We note that several approaches jointly estimate optical flow and depth by exploiting the cross-task constraints and consistency [40, 42, 43].”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, methodology): “For depth network, we train and test models on KITTI raw dataset [15] using Eigen [5]’s split that is the same with related works [10, 9, 11, 7].”
Monocular depth estimation based on deep learning: An overview (2020, background): “[89] design an additional network to estimate the optical flow.”