Generating Videos with Scene Dynamics

By Carl Vondrick, Hamed Pirsiavash, Antonio Torralba. 2016

1,581 citations recorded by the source corpus.

We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “To tackle the issues inherent in these models, most notably the tendency to generate blurry outputs due to restricted modeling power, inadequate prior distributions, or optimization of a lower bound in place of the true likelihood, these have been combined with adversarial objectives [26, 39, 23], h”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “Some use a single static stochastic variable, which is then “decoded” into a sequence using an RNN, either using a VAE-style loss [32, 35] or a GAN-style loss [31, 17].”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, background): “In order to overcome the bottleneck, there have been recent efforts on visual representation learning using web-supervision [2, 5, 6, 9, 21, 23, 24, 27] or unsupervised [10, 11, 31, 32, 34, 42, 43] paradigms.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Contrastive Multiview Coding (2019, neutral): “0 VGAN* [77] 2 52.”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “[46] proposed generative adversarial networks for video with a spatial-temporal convolutional architecture”
Video Pixel Networks (2016, background): “…models based on deep neural networks (Srivastava et al., 2015a; Oh et al., 2015), to models that predict quantized image patches (Ranzato et al., 2014), incorporate motion priors (Patraucean et al., 2015; Finn et al., 2016) or use adversarial losses (Mathieu et al., 2015; Vondrick et al., 2016).”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “While there have certainly been impressive efforts to model videos (Vondrick et al., 2016; Kalchbrenner et al., 2016; Tulyakov et al., 2018; Clark et al., 2019), high-fidelity natural videos is one notable modality that has not seen the same level of progress in generative modeling as compared to im”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “GANs enable a wide variety of applications such as image generation [1, 41, 60], representation learning [44], image manipulation [62], object detection [32], and video applications [37, 50, 52].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Recent methods adopt the same idea for conditional image generation applications, such as text2image [36], image inpainting [34], and future prediction [32], as well as to other domains like videos [50] and 3D models [53].”
Learning Latent Dynamics for Planning from Pixels (2018, background): “To obtain realistic predictions, Mathieu et al. (2015) and Vondrick et al. (2016) use adversarial losses.”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Most of the recent efforts on learning deep representations for videos have been focusing on capturing the appearance and local motion pattern from short video clips [22, 13, 11], in an supervised [26, 5, 35] or self-supervised [15, 31] fashion.”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “Recently, GANs have shown great potential in simulating complex data distributions, such as those of texts [6], images [32] and videos [48].”
Mode Regularized Generative Adversarial Networks (2016, methodology): “…image synthesis from text Reed et al. (2016) and edge map Isola et al. (2016), real-time image manipulation Zhu et al. (2016), temporal image generation Zhou & Berg (2016); Saito & Matsumoto (2016); Vondrick et al. (2016), texture synthesis, style transfer, and video stylization Li & Wand (2016).”
SoundNet: Learning Sound Representations from Unlabeled Video (2016, background): “Cross-Modal Learning and Unlabeled Video: Our approach is broadly inspired by efforts to model cross-modal relations [24, 14, 7, 26] and works that leverage large amounts of unlabeled video [25, 41, 8, 40, 39].”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “[46] learn a model of scene dynamics and use it to synthesize video sequences from single images.”
Variational Continual Learning (2017, background): “By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”