The Pose Knows: Video Forecasting by Generating Pose Futures

By Jacob Walker, Kenneth Marino, Abhinav Kumar Gupta, Martial Hebert. IEEE International Conference on Computer Vision, 2017

359 citations recorded by the source corpus.

Current approaches to video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is that forecasting needs to be done first at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene (humans) and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.

Subfields: Computer Vision and Pattern Recognition, Economics and Econometrics

View Paper: https://www.semanticscholar.org/paper/3df8cc0384814c3fb05c44e494ced947a7d43f36

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Economics and Econometrics

Citations in the corpus

Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “VAEs have been used for image and video generation [7, 13, 28, 29, 33, 41, 42, 44].”
QuaterNet: A Quaternion-based Recurrent Model for Human Motion (2018, methodology): “Pixellevel prediction using human pose as an intermediate variable has been explored [56, 59].”
Everybody Dance Now (2018, background): “Furthermore, [39, 38] have shown pose is an effective supervisory signal for future prediction and video generation.”
Adaptive Appearance Rendering (2021, methodology): “Simple generation networks can still produce blurry images as shown in [38].”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, background): “For this reason, many prediction-based approaches [57, 85, 84, 32, 46, 90] and other generative models [16, 78] leverage adversarial training [27] since the use of an ad-”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “[54] 2017 [208] vED [115], [116] RGB,Po RGB X `2, CE,KL,AL R ? ? ? X 7”
Deformable GANs for Pose-Based Human Image Generation (2017, background): “Generally speaking, U-Net based architectures are frequently adopted for pose-based person-image generation tasks [8, 12, 18, 20].”
Grounded Human-Object Interaction Hotspots From Video (2018, background): “Predicting future frames in videos has been studied extensively in computer vision [42, 35, 57, 34, 51, 54, 58, 39, 60, 56, 28].”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, methodology): “Sequence to sequence learning [38] has been used for different applications in computer vision such as representation learning [37], video captioning [40, 50], human pose estimation [44], motion prediction [20], or body pose labeling and forecasting [8, 44].”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, methodology): “Pose-GAN [121] takes a mixed VAE and GAN approach.”
How Generative Adversarial Networks and Their Variants Work (2017, methodology): “Pose-GAN [126] takes a VAE and GAN combining approach.”
Video-to-Video Synthesis (2018, background): “Conditioning on the observed frames, video prediction models are trained to predict future frames [15,18,36,41,42,45,50,65,68,71,72,77].”
r2p2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting (2018, background): “[42,47] also predict future video frames with an intermediate pose prediction.”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, background): “There are also works focusing on predicting future motions in image space [10, 58, 59, 9, 64, 25, 12].”
Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, background): “Recently, deep networks are playing increasingly crucial roles: some recurrent-network-based models generated future poses step-by-step [9, 20, 32, 41, 45, 11, 30, 12, 28]; some feed-forward networks [26, 31] tried to reduce error accumulation for stable prediction; imitation-learning algorithm was ”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, methodology): “608 Pose-Knows [68] 6.”
Natural and Effective Obfuscation by Head Inpainting (2017, methodology): “Prior work on structure-guided image generation has shown that such a guidance is indeed very helpful for generating images with complex inner structures (e.g. persons) [13, 5, 28, 30, 6, 33, 14, 2].”
Video Generation From Text (2017, background): “Moving to speciﬁc features such as human pose or skeleton generation could provide improvements to this issue (Chao et al. 2017; Walker et al. 2017).”
Sharpness-Aware Low-Dose CT Denoising Using Conditional Generative Adversarial Network (2017, methodology): “adopted a cGAN by conditioning on the predicted future pose information to synthesize future frames of a video [63].”