FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

By Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox. 2016

3,400 citations recorded by the source corpus.

The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a subnetwork specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “Ground-truth for this task is generated by the Flownet2 architecture [34], which is instead provided with the current and future frames I t , I t +1 .”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “In contrast, recent learning-based solutions to motion estimation [10, 15, 24], stereo matching [20, 21, 42] and 3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that are much more robust to local viewpoint and lighting changes.”
Tracking Emerges by Colorizing Videos (2018, methodology): “Unless otherwise stated, we use the best performing optical ﬂow based oﬀ FlowNet2 [23].”
Video-to-Video Synthesis (2018, methodology): “We use the optical ﬂow extracted by FlowNet2 [32] as the ground truth ﬂow w .”
Playing for Benchmarks (2017, background): “The poor performance of FlowNet is likely due to its exclusive training on a different dataset; we expect that training this model (or its successor [29]) on VIPER will yield much better results.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, background): “Although some recent approaches [11,10,13] have been proposed to leverage temporal consistency, they depend on models pretrained on other tasks such as optical flow [17,18] or motion segmentation [19], to extract temporal information.”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, background): “We have found current machine learning approaches [16, 22] to be limiting, in that they have not been generalized to arbitrary numbers of views (spatial or temporal); and available datasets [10, 17] are only stereo and low resolution.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, methodology): “If we replace it with the FlowNet2 propagation scheme, where masks are propagated using the optical flow predicted by a FlowNet2 network [20], the accuracy drops from 46.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “The best performing approach FlowNet2 (Ilg et al. (2016)) trains a deep neural network to solve the optical flow problem.”
Consistent video depth estimation (2020, methodology): “Specifically, we compute the stereo disparity by taking the horizontal components from the estimated flow on the stereo pair (using Flownet2 [Ilg et al. 2017]).”
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (2019, methodology): “However, these methods [6, 12, 35] are too complex, for instance, Wang et al.”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, neutral): “0 [17], DeepLab v3+ [7], Mask R-CNN [14], Re-ID network [23] and so on.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “On Sintel (final) dataset, our performance is only second to the recent FlowFormer [82] model, which uses a Transformer model that is pretrained on the large scale ImageNet dataset and is more computationally expensive Method FlowNet2 [16] PWC-Net+ [90] RAFT [21] GMFlow+”
Tracking Objects as Points (2020, methodology): “As an alternative motion model, we use FlowNet2 [15].”
Occlusion Aware Unsupervised Learning of Optical Flow (2017, methodology): “Since most deep networks are built to predict flow using two consecutive frames and trained with supervised learning [26], it would require a large amount of training data to obtain reasonably high accuracy [35].”