Occlusion Aware Unsupervised Learning of Optical Flow

By Yang Wang, Yezhou Yang, Zhenheng Yang, Liang Zhao, Wei Xu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

340 citations recorded by the source corpus.

It has been recently shown that a convolutional neural network can learn optical flow estimation with unsupervised learning. However, the performance of the unsupervised methods still has a relatively large gap compared to its supervised counterpart. Occlusion and large motion are some of the major factors that limit the current unsupervised learning of optical flow methods. In this work we introduce a new method which models occlusion explicitly and a new warping way that facilitates the learning of large motion. Our method shows promising results on Flying Chairs, MPI-Sintel and KITTI benchmark datasets. Especially on KITTI dataset where abundant unlabeled samples exist, our unsupervised method outperforms its counterpart trained with supervised learning.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/0b197f323e7d5514e5ada388858e38890dce0148

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Learning Correspondence From the Cycle-Consistency of Time (2019, methodology): “Recent work on optical flow estimation [43, 24, 68, 75, 45] also utilizes forward-backward consistency as an optimization goal.”
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders (2020, methodology): “Following [56], we define first-order derivatives of the saliency map in the smoothness term as”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, neutral): “Therefore, it usually requires multi-frame estimates [14], explicit reasoning about occlusions [42], hand-crafted loss terms [23, 25], or various data augmentation strategies [21].”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, methodology): “Therefore, the task is often simpliﬁed by deforming an initial mesh [73] or stitching multiple 3D patches [24].”
DDFlow: Learning Optical Flow with Unlabeled Data Distillation (2019, methodology): “2 from (Wang et al. 2018)).”
Self-supervised Video Representation Learning for Correspondence Flow (2019, positive): “The basic principles are based on brightness constancy and spatial smoothness [28, 43, 49].”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, background): “Multiple approaches disconnected from geometry have been proposed for handling occlusions in the context of optical flow [36, 15, 25].”
SelFlow: Self-Supervised Learning of Optical Flow (2019, methodology): “Recent studies propose to first obtain an occlusion map and then exclude those occluded pixels when computing the photometric difference [29, 46].”
Weakly-Supervised Salient Object Detection via Scribble Annotations (2020, background): “Inspired by the smoothness loss [9, 38], we also impose such constraint inside the salient regions.”
Learning to Estimate Hidden Motions with Global Motion Aggregation (2021, background): “In the deep learning literature, the occlusion problem has been addressed in an unsupervised learning setting [41, 21, 25], however, existing supervised learning approaches all rely on convolutions to interpolate in occluded regions, which are prone to failure for more significant occlusions.”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, methodology): “2d, we first perform bi-directional flow and occlusion estimation by simply switching the order of the input feature maps for the decoder [40, 58].”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “…Prominent examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement [7,…”
Digging Into Self-Supervised Monocular Depth Estimation (2018, background): “Learning based algorithms are also among some of the best performing for stereo estimation [72, 42, 60, 25] and optical flow [20, 63].”