Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness

By Jason J. Yu, Adam W. Harley, Konstantinos G. Derpanis. 2016

434 citations recorded by the source corpus.

Recently, convolutional networks (convnets) have proven useful for predicting optical flow. Much of this success is predicated on the availability of large datasets that require expensive and involved data acquisition and laborious labeling. To bypass these challenges, we propose an unsupervised approach (i.e., without leveraging groundtruth flow) to train a convnet end-to-end for predicting optical flow between two images. We use a loss function that combines a data term that measures photometric constancy over time with a spatial term that models the expected variation of flow across the image. Together these losses form a proxy measure for losses based on the groundtruth flow. Empirically, we show that a strong convnet baseline trained with the proposed unsupervised approach outperforms the same network trained with supervision on the KITTI dataset.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, neutral): “Following FlowNet, several papers have studied optical flow estimation with CNNs: featuring a 3D CNN [30], unsupervised learning [1, 33], carefully designed rotationally invariant architectures [28], or a pyramidal approach based on the coarse-to-fine idea of variational methods [20].”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, background): “There have also been several attempts at estimating optical ﬂow using un-supervised learning [3, 45].”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, methodology): “To alleviate the dependency on large-scale annotated datasets, several works have been proposed to exploit the classical assumptions of brightness constancy and spatial smoothness on the disparity map or the flow field [17,21,28,43,71].”
Softmax Splatting for Video Frame Interpolation (2020, methodology): “…examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical ﬂow prediction [32, 47, 52], optical ﬂow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement [7, 46,…”
Lucid Data Dreaming for Object Tracking (2017, methodology): “Instead of using a fixed model for all cases, it is also possible to obtain specialized perdataset models, either via self-supervision [61, 40, 64, 66] or by using the first frame annotation of each video in the dataset as training/tuning set.”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “To alleviate the need for training data with ground truth in a specific domain, unsupervised [1, 40, 47, 58, 63, 66] and semisupervised [31, 65] alternatives have also been developed.”
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction (2018, methodology): “An obvious extension to the above framework is to use structure-from-motion techniques to estimate the interframe motion (optic flow) [15] instead of depth using the known stereo geometry.”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, neutral): “[16] and Ren et al.”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “There is also a long line of work which trains directly on unlabelled data, using a variety of auxiliary objectives to encourage tracking to emerge [65, 60, 62, 29, 7].”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, background): “Instead of merely considering two frames [38], a memory module is proposed to keep the temporal information of a video sequence [21].”
Occlusion Aware Unsupervised Learning of Optical Flow (2017, methodology): “Table 1 summarizes the EPE of our method and previous state-of-the-art deep learning methods, including FlowNet [17], SpyNet [40], FlowNet2 [27], DSTFlow [41] and BackToBasic [31].”
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras (2018, methodology): “The self-supervised loss is modeled after frame based self-supervised ﬂow networks such as Yu et al. [28] and Meister et al. [16], where a photometric loss is used as a supervisory signal in place of direct supervision.”
Deep learning for physical processes: incorporating prior scientific knowledge (2017, methodology): “This result theoretically grounds the use of the STN for Optical Flow in many recent articles [Zhu et al., 2017], [Yu et al., 2016], [Patraucean et al., 2015], [Finn et al., 2016]: in equation 3, when D → 0, we recover the brightness constancy constraint equation, used in the latter.”
Weakly-supervised convolutional neural networks for multimodal image registration (2018, background): “The described additive displacement skip layers are more efficient to compute and, potentially, easier to train, comparing to composing displacements at different levels or concatenating warped input images (Ilg et al., 2017; Yu et al., 2016), both requiring resampling.”
Video Frame Synthesis Using Deep Voxel Flow (2017, neutral): “Basics [37] 9.”
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, methodology): “Unsupervised methods have tried to independently solve for optical flow [14, 24, 35] by minimizing a photometric loss.”
SelFlow: Self-Supervised Learning of Optical Flow (2019, methodology): “The basic principles are based on brightness constancy and spatial smoothness [20, 37].”