Learning to Generate Long-term Future via Hierarchical Prediction

By Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, Honglak Lee. 2017

400 citations recorded by the source corpus.

We propose a hierarchical approach for making long-term predictions of future frames. To avoid inherent compounding errors in recursive pixel-level prediction, we propose to first estimate high-level structure in the input frames, then predict how that structure evolves in the future, and finally by observing a single frame from the past and the predicted high-level structure, we construct the future frames without having to observe any of the pixel-level predictions. Long-term video prediction is difficult to perform by recurrently observing the predicted frames because the small errors in pixel space exponentially amplify as predictions are made deeper into the future. Our approach prevents pixel-level error propagation from happening by removing the need to observe the predicted frames. Our model is built with a combination of LSTM and analogy based encoder-decoder convolutional neural networks, which independently predict the video structure and generate the future frames, respectively. In experiments, our model is evaluated on the Human3.6M and Penn Action datasets on the task of long-term pixel-level video prediction of humans performing actions and demonstrate significantly better results than the state-of-the-art.

Subfields: Computer Vision and Pattern Recognition, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Time-Series & Forecasting

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Therefore, we take inspiration from Amos et al. (2018) and earlier related ideas (Chiappa et al., 2017; Villegas et al., 2017; Lamb et al., 2016), and train all multi-step predictions of the model.”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, background): “[29] predicted human joints for future frame prediction.”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Recent image-conditioned models have shown promising results for numerous image-to-image translation tasks such as maps → satellite images, sketches → photos, labels → images [12, 41, 40], future frame prediction [32], superresolution [17], and inpainting [36].”
Prior Networks for Detection of Adversarial Attacks (2018, background): “Neural Networks (NNs) have become the dominant approach to addressing computer vision (CV) [1, 2, 3], natural language processing (NLP) [4, 5, 6], speech recognition (ASR) [7, 8] and bioinformatics [9, 10] tasks.”
Predictive Uncertainty Estimation via Prior Networks (2018, background): “Neural Networks (NNs) have become the dominant approach to addressing computer vision (CV) [1, 2, 3], natural language processing (NLP) [4, 5, 6], speech recognition (ASR) [7, 8] and bio-informatics (BI) [9, 10] tasks.”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, background): “l networks (DNNs) have demonstrated state-of-the-art performance on many classiﬁ- cation tasks, e.g., speech recognition (Hannun et al., 2014), image classiﬁcation (Girshick, 2015), video prediction (Villegas et al., 2017) and medical diagnosis (Caruana et al., 2015). Even though DNNs achieve high a”
Stochastic Video Generation with a Learned Prior (2018, background): “Villegas et al. (2017b) propose a hierarchical model that ﬁrst generates high level structure of a video and then generates pixels conditioned on the high level structure.”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “Recent common approaches for full frame prediction predict the feature representations that generate future frames [6, 22, 23, 37, 38, 39] in a sequence-to-sequence framework [4, 32], which has been extended to incorporate spatio-temporal recurrence [15, 31, 43].”
Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “[41] introduced a hierarchical video prediction model consisting of two stages: it first predicts the motion of a set of landmarks using an LSTM, then generates images from the landmarks.”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “Predicting future frames in videos has been studied extensively in computer vision [42, 35, 57, 34, 51, 54, 58, 39, 60, 56, 28].”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, background): “[46] also generate future frames by extracting a representation of appearance and human pose, but, differently from us, require ground-truth pose annotations.”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, background): “Predicting future frames, poses or image segmentations in videos has been studied in several works [20, 16, 13, 31, 11, 27, 18, 29, 33, 32, 17, 12].”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “This shows that PhyDNet performs similarly than [9] for the prediction horizon considered, without requiring additional human pose annotations.”
Recycle-GAN: Unsupervised Video Retargeting (2018, background): “Recent work [16,44,46] used temporal models to predict long term future poses from a single 2D image.”
Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction (2018, methodology): “…been devised for deep learning machines that conduct video prediction (Mathieu et al., 2015; Srivastava et al., 2015; Lotter et al., 2016; Vondrick et al., 2016; Villegas et al., 2017a,b), including CNN-LSTM Encoder-Decoder used as a control in the original paper of PredNet (Lotter et al., 2016).”
Everybody Dance Now (2018, background): “Furthermore, [39, 38] have shown pose is an effective supervisory signal for future prediction and video generation.”
r2p2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting (2018, background): “[42,47] also predict future video frames with an intermediate pose prediction.”