Deep multi-scale video prediction beyond mean square error

By Michaël Mathieu, Camille Couprie, Yann LeCun. 2015

2,004 citations recorded by the source corpus.

Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectories. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “To tackle the issues inherent in these models, most notably the tendency to generate blurry outputs due to restricted modeling power, inadequate prior distributions, or optimization of a lower bound in place of the true likelihood, these have been combined with adversarial objectives [26, 39, 23], h”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “Some use a single static stochastic variable, which is then “decoded” into a sequence using an RNN, either using a VAE-style loss [32, 35] or a GAN-style loss [31, 17].”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Video Pixel Networks (2016, background): “…models based on deep neural networks (Srivastava et al., 2015a; Oh et al., 2015), to models that predict quantized image patches (Ranzato et al., 2014), incorporate motion priors (Patraucean et al., 2015; Finn et al., 2016) or use adversarial losses (Mathieu et al., 2015; Vondrick et al., 2016).”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “…widely used in many areas for its strong ability of modeling and learning approximately true dataset distributions, including image generation [30]–[32], representation learning [33], image retrieval [34], [35], object detection [36], video applications [37], [38] and image translation [39]–[45].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “The image-conditional models have tackled image prediction from a normal map [52], future frame prediction [37], product photo generation [56], and image generation from sparse annotations [28, 45] (c.”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, methodology): “[17] used GANs for predicting future frames in videos by conditioning on previous frames.”
Generative adversarial networks (2014, neutral): “GANs are thus great for learning in situations where there are many possible correct answers, such as predicting the many possible futures that can happen in video generation.(19) GANs and GAN-like models can be used to learn to transform data from one domain into data from another domain, even with”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “GANs enable a wide variety of applications such as image generation [1, 41, 60], representation learning [44], image manipulation [62], object detection [32], and video applications [37, 50, 52].”
Gradient descent GAN optimization is locally stable (2017, methodology): “…deployed for tasks such as photo super-resolution, object generation, video prediction, language modeling, vocal synthesis, and semi-supervised learning, amongst many others [Ledig et al., 2016, Wu et al., 2016, Mathieu et al., 2015, Nguyen et al., 2016, Denton et al., 2015, Im et al., 2016].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Recent methods adopt the same idea for conditional image generation applications, such as text2image [39], image inpainting [36], and future prediction [34], as well as to other domains like videos [53] and 3D data [56].”
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, background): “Pixel-wise loss functions such as MSE struggle to handle the uncertainty inherent in recovering lost high-frequency details such as texture: minimizing MSE encourages finding pixel-wise averages of plausible solutions which are typically overly-smooth and thus have poor perceptual quality [35, 26].”
Generating Videos with Scene Dynamics (2016, background): “[33, 22] can generate video, but they require multiple input frames and empirically become blurry after extrapolating many frames.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, background): “Early works like (Mathieu et al., 2015; Srivastava et al., 2015; Han et al., 2019) aim to synthetic video frames through the image patches.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
Rewriting a Deep Generative Model (2020, background): “At a practical level, deep generative models are increasingly useful for image and video synthesis [57,86,35,12].”
Learning Latent Dynamics for Planning from Pixels (2018, background): “To obtain realistic predictions, Mathieu et al. (2015) and Vondrick et al. (2016) use adversarial losses.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “, generating images conditioned on labels [34, 37] or next frames of a video given a context window [33].”
Energy-based Generative Adversarial Network (2016, background): “Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to signiﬁcant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”