Unsupervised Deep Learning for Optical Flow Estimation

By Zhe Ren, Junchi Yan, Bingbing Ni, Xiaokang Yang, Hongyuan Zha. 2017

325 citations recorded by the source corpus.

Recent work has shown that optical flow estimation can be formulated as a supervised learning problem. Moreover, convolutional networks have been successfully applied to this task. However, supervised flow learning is obfuscated by the shortage of labeled training data. As a consequence, existing methods have to turn to large synthetic datasets for easily computer generated ground truth. In this work, we explore if a deep network for flow estimation can be trained without supervision. Using image warping by the estimated flow, we devise a simple yet effective unsupervised method for learning optical flow, by directly minimizing photometric consistency. We demonstrate that a flow network can be trained from end-to-end using our unsupervised scheme. In some cases, our results come tantalizingly close to the performance of methods trained with full supervision.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occlusion Aware Unsupervised Learning of Optical Flow (2017, methodology): “Our ﬂow results are also free from the artifacts seen in DSTFlow (see [41] Figure 4c) in the occlusion area.”
Unsupervised End-to-end Learning for Deformable Medical Image Registration (2017, background): “Several STN-based approaches have been proposed to address similar problems in natural scenes, such as optical flow estimation [23], [24] depth estimation [25], [26] and single-view reconstruction [27].”
SelFlow: Self-Supervised Learning of Optical Flow (2019, methodology): “9 – – – DSTFlow+ft [37] (6.”
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss (2017, methodology): “This limitation is shared by previous works employing an unsupervised proxy loss (Ahmadi and Patras 2016; Yu, Harley, and Der-panis 2016; Ren et al. 2017).”
DDFlow: Learning Optical Flow with Unlabeled Data Distillation (2019, background): “We compare our results with state-of-the-art unsupervised methods including BackToBasic(Jason, Harley, and Derpanis 2016), DSTFlow(Ren et al. 2017), UnFlow(Meister, Hur, and Roth 2018), OccAwareFlow(Wang et al.”
Deep Nets: What have They Ever Done for Vision? (2018, background): “from motion can be learned without requiring detailed supervision but only an energy function model (Ren et al., 2017; Zhou et al., 2017).”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “To alleviate the need for training data with ground truth in a specific domain, unsupervised [1, 40, 47, 58, 63, 66] and semisupervised [31, 65] alternatives have also been developed.”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, methodology): “(1), and adding flow smoothness term yields unsupervised learning of optical flow [7], [17].”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, unknown): “[37] and Yu et al.”
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras (2018, methodology): “[20] train an optical flow prediction network using the traditional brightness constancy and smoothness constraints developed in optimization based methods such as the Lucas Kanade method Lucas et al.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, background): “he dynamic object problem in self-supervised monocular depth estimation. Most existing works follow the classical way in considering optical ow, which can also be predicted in an unsupervised fashion [36,50]. By simultaneously predicting optical ow and depth, existing works impose losses for cross-t”