UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

By Simon Meister, Junhwa Hur, Stefan Roth. 2017

637 citations recorded by the source corpus.

In the era of end-to-end deep learning, many advances in computer vision are driven by large amounts of labeled data. In the optical flow setting, however, obtaining dense per-pixel ground truth for real scenes is difficult and thus such data is rare. Therefore, recent end-to-end convolutional networks for optical flow rely on synthetic datasets for supervision, but the domain mismatch between training and test scenarios continues to be a challenge. Inspired by classical energy-based optical flow methods, we design an unsupervised loss based on occlusion-aware bidirectional flow estimation and the robust census transform to circumvent the need for ground truth flow. On the KITTI benchmarks, our unsupervised approach outperforms previous unsupervised deep networks by a large margin, and is even more accurate than similar supervised methods trained on synthetic datasets alone. By optionally fine-tuning on the KITTI training data, our method achieves competitive optical flow accuracy on the KITTI 2012 and 2015 benchmarks, thus in addition enabling generic pre-training of supervised networks for datasets with limited amounts of ground truth.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “4 Our approach is similar to the optic ﬂow method proposed by [43].”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, methodology): “[17] exploited the cycle consistency in unsupervised optical flow estimation by designing a bidirectional consensus loss during training.”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, background): “…advancements, for instance Spatial Transformer Networks [10], have indeed opened the door to a variety of differentiable geometric constraints used as learning objectives capturing key scene properties characterizing optical ﬂow [13], [24], depth [6], [5], [25], [16], and camera pose [7], [25].”
Softmax Splatting for Video Frame Interpolation (2020, background): “(cid:1) Prominent examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement…”
Recycle-GAN: Unsupervised Video Retargeting (2018, methodology): “More constraints using auxiliary tasks from the visual data itself (as used for other vision tasks [33,50]) could help in learning better transformation models.”
Generating Liquid Simulations with Deformation-aware Neural Networks (2017, methodology): “Thus, our method shares similarities with unsupervised approaches for optical flow [20, 29].”
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras (2018, methodology): “[13] extend the loss with occlusion masks and a second order smoothness term, and Lai et al.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, positive): “Mehtod SubCategory Code Contribution VideoGAN [85] Generation 3 Forerunner of video generation with GAN MocoGAN [86] Generation 3 Decomposing motion and content for video generation with GAN TemporalGAN [144] Generation 3 Decomposing temporal and image generator for video generation Video Colorizati”
SelFlow: Self-Supervised Learning of Optical Flow (2019, methodology): “Recent studies propose to first obtain an occlusion map and then exclude those occluded pixels when computing the photometric difference [29, 46].”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “[30] also independently introduce a bidirectional census loss.”
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, methodology): “Unsupervised methods have tried to independently solve for optical flow [14, 24, 35] by minimizing a photometric loss.”
GMFlow: Learning Optical Flow via Global Matching (2021, methodology): “The bidirectional flow can be used for occlusion detection with forward-backward consistency check (following [28]), as shown in Fig.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “Note that during training we only predict unidirectional flow while at inference, we can obtain bidirectional flow for free, without requiring to forward the network twice, unlike previous regression-based methods [44], [84].”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “In motion analysis, forward-backward consistency check can be used for detecting occlusion [28, 20, 52] or learning visual correspondence [45].”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, methodology): “2d, we first perform bi-directional flow and occlusion estimation by simply switching the order of the input feature maps for the decoder [40, 58].”
SegStereo: Exploiting Semantic Information for Disparity Estimation (2018, background): “[32] defined a bidirectional census loss to train optical flow.”