FlowNet: Learning Optical Flow with Convolutional Networks

By Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox. 2015

4,664 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “Tasks include region proposals [31], contour detection [32], depth regression [33], optical flow [34], and weaklysupervised semantic segmentation [35], [36], [37], [38].”
Recurrent Instance Segmentation (2015, background): “Problems with these characteristics, such as semantic segmentation [12,14] and optical flow [35], are often tackled using structures based on convolutions, in which the intermediate representations of the images preserve the spatial information.”
Learning to Refine Object Segments (2016, methodology): “Finally, [39] proposed to progressively increase the resolution of an optical ﬂow map.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “As hardware acceleration grew more powerful and neural nets proved themselves in image understanding [41, 101, 78], interest intensified: autoregressive models were developed that could take samples of handwriting and complete them [28], and new convolutional network designs led to good results in s”
Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “The feature calibration first computes the spatial correlation (Dosovitskiy et al., 2015) between Fq and Fr.”
Dynamic Filter Networks (2016, background): “Though the ﬂow map is not that smooth, this byproduct is learned in an unsupervised way by only training on unlabeled video data which is different from e.g. [4].”
Variational Information Distillation for Knowledge Transfer (2019, background): “, depth estimation [8], pose estimation [26], optical flow [7], object classification [11], detection [10], and segmentation [25].”
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks (2018, background): “However, both [4] and [17] demonstrate that complete realism is not necessary to improve results with synthetic data.”
Synthetic Data for Text Localisation in Natural Images (2016, methodology): “[9] use floating chair renderings to train dense optical flow regression networks.”
Geometric Deep Learning: Going beyond Euclidean data (2016, background): “Typical computer vision applications requiring covariance are semantic image segmentation [14] or motion estimation [57].”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, methodology): “Encoder-decoder architectures with an up-convolutional part have been used successfully for semantic segmentation tasks [23, 22, 30, 3, 27], depth estimation [20] and optical ﬂow estimation [7].”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, methodology): “We will make our library and wrapper source code available.”
OctNetFusion: Learning Depth Fusion from Data (2017, methodology): “As the 3D input to our method can be sparse and incomplete, we refrain from using the classical U-shaped architecture as common for 2D-to-2D prediction tasks [1, 13].”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, positive): “Up-convolutional1 decoder architectures have become a standard tool for tasks requiring image generation [9, 27, 20] or per-pixel prediction [25, 8].”
Lightweight Probabilistic Deep Networks (2018, methodology): “[10, 15, 37].”