Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

By Tianfan Xue, Jiajun Wu, Katherine Louise Bouman, Bill Freeman. 2016

418 citations recorded by the source corpus.

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Videos with Scene Dynamics (2016, background): “Our work is also related to efforts to predict the future in video [33, 22, 43, 50, 42, 17, 8, 54] as well as concurrent work in future generation [6, 15, 20, 49, 55].”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “[48] develop a probabilistic model that enables synthe-”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Deep generative models with structured bottlenecks have recently seen a lot of advances [3, 19, 36, 38, 12] but they do not explicitly reason about geometry.”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “Some use a single static stochastic variable, as in a conditional VAE (Kingma & Welling, 2014), which is then “decoded” into a sequence using an RNN, either using a VAE-style loss (Walker et al., 2016; Xue et al., 2016) or a GAN-style loss (Vondrick et al.”
Deep visual foresight for planning robot motion (2016, background): “Neural network-based video prediction is still in its early stages, with most state-of-the-art methods making accurate predictions only a few frames into the future [5].”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “[29] predicts transformations applied to latent feature maps, rather than the image itself, but only demonstrates single frame prediction.”
Unsupervised Learning of Disentangled Representations from Video (2017, background): “Other works predict optical ﬂows ﬁelds that can be used to extrapolate motion beyond the current frame, e.g. [16, 38, 33].”
Toward Multimodal Image-to-Image Translation (2017, background): “While conditional VAEs [42] and autoregressive models [32, 33] have shown promise [16, 44, 46], imageto-image conditional GANs have lead to a substantial boost in the quality of the results.”
Stochastic Variational Video Prediction (2017, background): “Xue et al. (2016) predicts a single stochastic frame using cross convolutional networks in a VAE-like architecture.”
Stochastic Video Generation with a Learned Prior (2018, background): “Other works predict optical ﬂows ﬁelds that can be used to extrapolate motion beyond the current frame, e.g. ( Liu, 2009; Xue et al., 2016; Walker et al., 2015).”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, methodology): “-frame video prediction (Ranzato et al., 2014; Michalski et al., 2014; Srivastava et al., 2015; Mathieu et al., 2016; Lotter et al., 2015; Patraucean et al., 2015; Oh et al., 2015; Finn et al., 2016; Xue et al., 2016; Vondrick et al., 2016; Brabandere et al., 2016), but we take particular inspiratio”
Stochastic Adversarial Video Prediction (2018, methodology): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Learning to Generate Long-term Future via Hierarchical Prediction (2017, background): “Xue et al. (2016) proposed a probabilistic model for predicting possible mo- tions of a single input frame by training a motion encoder in a variational autoencoder approach.”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, neutral): “Xue et al. (2016) have approached this problem by designing a variational autoencoder architecture that uses the latent stochastic units of the VAE to make choices about the direction of motion of objects, and generates future frames conditioned on these choices.”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “Visual prediction is a core problem in computer vision that has been studied in several contexts, including activity prediction and early recognition [20, 30], human pose and trajectory forecasting [1, 18], and future frame prediction [22, 31, 39, 44].”
Iterative Amortized Inference (2018, background): “…et al., 1998) has been essential in learning deep directed latent variable models on high-dimensional data, enabling extraction of complex, non-linear relationships, such as object identities (Higgins et al., 2016) and dynamics (Xue et al., 2016; Karl et al., 2017) directly from observations.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Note that applying the estimated kernels to the input images is a lo-cal convolution and we implement it as a network layer of our neural network similar to a position-varying dynamics convolution layer in recent work [14, 21, 58]; therefore our neural network is end-to-end trainable.”
Decomposing Motion and Content for Natural Video Sequence Prediction (2017, background): “In terms of the way data is observed, the closest work to ours is Xue et al. (2016).”