Dense Optical Flow Prediction from a Static Image

By Jacob Walker, Abhinav Kumar Gupta, Martial Hebert. 2015

220 citations recorded by the source corpus.

Given a scene, what is going to move, and in what direction will it move? Such a question could be considered a non-semantic form of action prediction. In this work, we present a convolutional neural network (CNN) based approach for motion prediction. Given a static image, this CNN predicts the future motion of each and every pixel in the image in terms of optical flow. Our CNN model leverages the data in tens of thousands of realistic videos to train our model. Our method relies on absolutely no human labeling and is able to predict motion based on the context of the scene. Because our CNN model makes no assumptions about the underlying scene, it can predict future optical flow on a diverse set of scenarios. We outperform all previous approaches by large margins.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-task Self-Supervised Visual Learning (2017, background): “In the last few years, a considerable number of such tasks have been proposed [1, 2, 6, 7, 8, 17, 20, 21, 23, 25, 26, 27, 28, 29, 31, 39, 40, 42, 43, 46, 47], such as asking a neural network to colorize grayscale images, fill in image holes, solve jigsaw puzzles made from image patches, or predict m”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “There are two common approaches for unsupervised learning: (a) using a discriminative framework with auxiliary tasks where supervision comes for free, such as context prediction [1,2] or temporal embedding [3,4,5,6,7,8]; (b) using a generative framework where the underlying model is compositional an”
SoundNet: Learning Sound Representations from Unlabeled Video (2016, background): “Cross-Modal Learning and Unlabeled Video: Our approach is broadly inspired by efforts to model cross-modal relations [24, 14, 7, 26] and works that leverage large amounts of unlabeled video [25, 41, 8, 40, 39].”
Unsupervised Learning of Long-Term Motion Dynamics for Videos (2017, methodology): “We also report the prediction error of using a single input frame to predict the next frame similar to [33] (blue dot).”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “This approach differs from recent work on optic flow prediction [22, 14], which predicts where pixels will move to.”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, methodology): “For the optical flow baseline [4], we can take samples from the underlying softmax probability distribution.”
Stochastic Video Generation with a Learned Prior (2018, background): “Other works predict optical ﬂows ﬁelds that can be used to extrapolate motion beyond the current frame, e.g. ( Liu, 2009; Xue et al., 2016; Walker et al., 2015).”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, methodology): “A similar observation has been made by [38,37] that formulate a continuous variable estimation problem as classification.”
Self-Supervised Visual Planning with Temporal Skip Connections (2017, methodology): “Furthermore, unlike prior prediction methods [4, 5] our model does not require any additional, external systems for providing point-to-point correspondences between frames of different time-steps.”
The Curious Robot: Learning Visual Representations via Physical Interactions (2016, background): “Finally, our approach is also related to other eﬀorts which train deep networks using supervision from other sensors such as Kinect [11,12] or motion information such as optical ﬂow [13] since these approaches require manual labeling.”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, background): “Several recent papers [5, 41, 14] have trained unsupervised image representations targeting specific narrow task s.”
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks (2016, background): “For example, given an image of a car in front of a building, many would predict that the car is moving and the building is ﬁxed.”
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images (2015, methodology): “Predictive-CNN [35] 1.53 N 3 (ours) 1.29 Force and velocity estimation.”
Stochastic Adversarial Video Prediction (2018, neutral): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Learning 3D Human Dynamics From Video (2018, unknown): “Figure 2 illustrates the framework.”
Flow-Grounded Spatial-Temporal Video Prediction from Still Images (2018, methodology): “Similar to [14,43], we employ an existing optical flow estimator SPyNet [41] to obtain flows between GT frames for training the 3D-cVAE.”
The Pose Knows: Video Forecasting by Generating Pose Futures (2017, methodology): “Models that either try to directly forecast pixels [27, 33, 42, 47, 54] or pixel motion [9,30,48,50,55] are forced to perform all of these tasks simultaneously.”
Forecasting Interactive Dynamics of Pedestrians with Fictitious Play (2016, background): “There has been growing interest in developing computational models of human activities that can extrapolate unseen information and predict future unobserved activities [34, 38, 22, 33, 18, 42, 40, 37, 15, 14, 16, 17, 35, 9, 49, 43, 36, 32].”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “Several methods have been proposed for video prediction problem, which in high level can be categorized as flowbased methods where methods model motion transformation via flows [22], [29], [37], [39] and latent-representation based methods [11], [12], [19], [31], [42].”