Decomposing Motion and Content for Natural Video Sequence Prediction

By Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, Honglak Lee. 2017

639 citations recorded by the source corpus.

We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
Learning-based Video Motion Magnification (2018, background): “For frame extrapolation, there is a line of recent work [15,16,17] that directly synthesizes RGB pixel values to predict dynamic video frames in the future, but their results are often blurry.”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Somewhat related, Villegas et al. [70], Denton and Birodkar [71], Fraccaro et al. [72] propose autoencoder models for video sequence prediction with separate encoders disentangling the latent code into pose and content.”
Stochastic Video Generation with a Learned Prior (2018, background): “Villegas et al. (2017b) propose a hierarchical model that ﬁrst generates high level structure of a video and then generates pixels conditioned on the high level structure.”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, methodology): “We conducted a user study comparing our method with two state-of-the-art approaches: a Conditional-VGAN (C-VGAN) and Motion Content Network (MCNET) [40].”
Disentangled Sequential Autoencoder (2018, methodology): “For GAN-like models, both Villegas et al. (2017) and Den-ton & Birodkar (2017) proposed an auto-encoder architecture for next frame prediction, with two separate encoders responsible for content and pose information at each time step.”
Stochastic Adversarial Video Prediction (2018, background): “A number of prior works have used adversarial losses for deterministic video prediction [7,11,12,13,40,41].”
Learning to Generate Long-term Future via Hierarchical Prediction (2017, methodology): “We evaluate the frames generated by our method using PSNR as measure, and separated the test data based on amount of motion, as suggested by Villegas et al. (2017).”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “Similarly, MCNet [39] disentangles motion from content using image differences and shared a single content vector in prediction.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Early video generation approaches relied primarily on generative adversarial networks (GANs) [18], either using convolutional networks to generate entire videos in parallel [5, 68, 82] or employing recurrent architectures to produce frames sequentially [14, 44, 49, 77, 81].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[91] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee.”
Improved Conditional VRNNs for Video Prediction (2019, background): “Other works have proposed to disentangle the motion and context of the frames to generate [37, 35, 6].”
Future Frame Prediction for Anomaly Detection - A New Baseline (2017, methodology): “[37] uses a LSTM based motion encoder to encode all history motion for prediction.”
Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks (2019, neutral): “video-analysis tasks such as anomaly detection [62], sequence prediction [63], and passenger-demand prediction [64], [65].”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, methodology): “In addition, LSTM can also be used to learn the visual features from videos especially to model the temporal information among frames [37, 40].”