Dual Motion GAN for Future-Flow Embedded Video Prediction

By Xiaodan Liang, Lisa Lee, Wei Dai, Eric P. Xing. IEEE International Conference on Computer Vision, 2017

375 citations recorded by the source corpus.

Future frame prediction in videos is a promising avenue for unsupervised video representation learning. Video frames are naturally generated by the inherent pixel flows from preceding frames based on the appearance and motion dynamics in the video. However, existing methods focus on directly hallucinating pixel values, resulting in blurry predictions. In this paper, we develop a dual motion Generative Adversarial Net (GAN) architecture, which learns to explicitly enforce future-frame predictions to be consistent with the pixel-wise flows in the video through a duallearning mechanism. The primal future-frame prediction and dual future-flow prediction form a closed loop, generating informative feedback signals to each other for better video prediction. To make both synthesized future frames and flows indistinguishable from reality, a dual adversarial training method is proposed to ensure that the futureflow prediction is able to help infer realistic future-frames, while the future-frame prediction in turn leads to realistic optical flows. Our dual motion GAN also handles natural motion uncertainty in different pixel locations with a new probabilistic motion encoder, which is based on variational autoencoders. Extensive experiments demonstrate that the proposed dual motion GAN significantly outperforms stateof-the-art approaches on synthesizing new video frames and predicting future flows. Our model generalizes well across diverse visual scenes and shows superiority in unsupervised video representation learning.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eb911796464b1e01e98c2b3d007a4ef9310272e2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Future Event Prediction: If and When (2019, methodology): “[12] then use dual motion GAN to predict future frames given an input video sequence.”
Flow-Grounded Spatial-Temporal Video Prediction from Still Images (2018, background): “Much progress has been made in the generated quality of future outputs by designing different network structures [15,16,2,17,18] or using different learning techniques, including adversarial loss [19,20], motion/content separation [4,21,5], and transformation parameters [22,23].”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, background): “Conditional generative adversarial networks(cGANs) have shown impressive results on image-to-image translation, whose goal is to translate an input image from one domain to another domain [12,38,5,34,18,19,35].”
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers (2018, background): “Some studies introduce adversarial training to avoid generating a blurred image caused by the image reconstruction loss [31, 27, 25].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Early works leverage deterministic methods to predict the most likely next frames [42–47].”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “Following [37, 45, 73], we center-crop Sixteen baselines are selected for comparision, including BeyondMSE [40], MCnet [62], DVF [35], Dual-GAN [32],”
PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction (2019, background): “In the computer vision community there is a large body of works dedicated to video and action prediction [20, 19, 24, 21, 7, 5, 17].”
AmbientGAN: Generative models from lossy measurements (2018, background): “…framework has been shown to be extremely powerful in modeling complex data distributions such as images [Radford et al. (2015); Arjovsky et al. (2017); Berthelot et al. (2017)], video [Liang et al. (2017); Vondrick et al. (2016)], and 3D models [Achlioptas et al. (2017); Wu et al. (2016)].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have also been used in other video applications such as video prediction [84], [351], [352] and video retargeting [353].”
Few-shot Video-to-Video Synthesis (2019, background): “Video generative models can be divided into three main categories, including 1) unconditional video synthesis models [54, 45, 51], which convert random noise samples to video clips, 2) future video prediction models [48, 24, 11, 34, 33, 63, 55, 56, 10, 53, 29, 27, 18, 28, 16, 40], which generate fut”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, methodology): “[14] train optical flow via video frame extrapolation, but their training uses the flow estimated by the EpicFlow method [23] as an additional supervision signal.”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “To constrain the challenging generation of high dimensional images, several methods rather predict geometric transformations between frames [16, 24, 75] or use optical flow [46, 38, 33, 32, 31].”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “It covers a wide spectrum of topics, including predicting future frames (Vondrick et al., 2016b; Liang et al., 2017), future action labels (Vondrick et al., 2016a; Gao et al., 2017b; Shi et al., 2018; Epstein et al., 2020), future human motions (Fragkiadaki et al., 2015; Mao et al., 2019), etc.”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “Predicting future frames in videos has been studied extensively in computer vision [42, 35, 57, 34, 51, 54, 58, 39, 60, 56, 28].”