Flow-Grounded Spatial-Temporal Video Prediction from Still Images

By Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang. 2018

142 citations recorded by the source corpus.

Existing video prediction methods mainly rely on observing multiple historical frames or focus on predicting the next one-frame. In this work, we study the problem of generating consecutive multiple future frames by observing one single still image only. We formulate the multi-frame prediction task as a multiple time step flow (multi-flow) prediction phase followed by a flow-to-frame synthesis phase. The multi-flow prediction is modeled in a variational probabilistic manner with spatial-temporal relationships learned through 3D convolutions. The flow-to-frame synthesis is modeled as a generative process in order to keep the predicted results lying closer to the manifold shape of real video sequence. Such a two-phase design prevents the model from directly looking at the high-dimensional pixel space of the frame sequence and is demonstrated to be more effective in predicting better and diverse results. Extensive experimental results on videos with different types of motion show that the proposed algorithm performs favorably against existing methods in terms of quality, diversity and human perceptual evaluation.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, neutral): “In recent years, GANs have been applied to numerous tasks ranging from image translation [3], [4], [5], image manipulation [6], [7], [8] to image restoration [9], [10], [11], [12], [13].”
Predicting Video with VQVAE (2021, background): “Others stepped aside from the problem of full pixel prediction and instead predicted pixel motion [13, 55, 19] or a decomposition of pixels and motion [8, 14, 18, 16, 24, 50, 52].”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, background): “Besides action recognition, other tasks of action understanding, including action detection and localization [14, 47, 54, 21, 16, 36], action segmentation [26, 9], and action generation [28, 41], also attract many researchers.”
Unsupervised Discovery of Parts, Structure, and Dynamics (2019, methodology): “We compare with 3DcVAE (Li et al., 2018), which takes one frame as input and predicts the next 16 frames.”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “To constrain the challenging generation of high dimensional images, several methods rather predict geometric transformations between frames [16, 24, 75] or use optical flow [46, 38, 33, 32, 31].”
Few-shot Video-to-Video Synthesis (2019, background): “…45, 51], which convert random noise samples to video clips, 2) future video prediction models [48, 24, 11, 34, 33, 63, 55, 56, 10, 53, 29, 27, 18, 28, 16, 40], which generate future video frames based on the observed ones, and 3) vid2vid models [57, 7, 12, 67], which convert semantic input…”
DeepLandscape: Adversarial Modeling of Landscape Video (2020, methodology): “Our approach is thus opposed to methods that animate landscapes and textures by generating warping fields applied to the raw pixels of the input static image [2, 19, 11, 20, 21].”
Learning 3D Human Dynamics From Video (2018, background): “There are several approaches that predict future from a single image [49, 53, 11, 29, 19], but all approaches predict future in 2D domains, while in this work we propose a framework that predicts 3D motions.”
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers (2018, background): “Another well-known approach is to predict intermediate features of videos such as optical flow [27, 29, 15, 26].”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, methodology): “Among these models, techniques based on variational autoencoders which optimize an evidence lower bound on the log-likelihood have been explored most widely (Babaeizadeh et al., 2017; Denton & Fergus, 2018; Lee et al., 2018; Xue et al., 2016; Li et al., 2018).”
Channel Attention Is All You Need for Video Frame Interpolation (2020, background): “…2017; Simonyan and Zis-serman 2014a; Xie et al. 2018), object detection and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al. 2018; Reda et al. 2018), and others (Chen et al. 2017; Lai et al. 2018)—frequently depend on the accurate optical ﬂow estimation since it…”