Learning 3D Human Dynamics From Video

By Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, Jitendra Malik. Computer Vision and Pattern Recognition, 2018

610 citations recorded by the source corpus.

From an image of a person in action, we can easily guess the 3D motion of the person in the immediate past and future. This is because we have a mental model of 3D human dynamics that we have acquired from observing visual sequences of humans in motion. We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features. At test time, from video, the learned temporal representation give rise to smooth 3D mesh predictions. From a single image, our model can recover the current 3D mesh as well as its 3D past and future motion. Our approach is designed so it can learn from videos with 2D pose annotations in a semi-supervised manner. Though annotated data is always limited, there are millions of videos uploaded daily on the Internet. In this work, we harvest this Internet-scale source of unlabeled data by training our model on unlabeled video with pseudo-ground truth 2D pose obtained from an off-the-shelf 2D pose detector. Our experiments show that adding more videos with pseudo-ground truth 2D pose monotonically improves 3D prediction performance. We evaluate our model on the recent challenging dataset of 3D Poses in the Wild and obtain state-of-the-art performance on the 3D prediction task without any fine-tuning. The project website with video can be found at https://akanazawa.github.io/human_dynamics/.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/436142577b23f89832fe6ee2d77ee9585e103729

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “In case of video inputs, temporal constraints have been used for body motion estimation in the context of neural networks [20, 24], or optimization [1, 51].”
C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “Besides the fully supervised methods [25, 26], several works have explored multi-view supervision [20, 29, 31], ordinal depth supervision [28], unpaired 2D-3D data [30, 36, 41, 15] or videos [17] to alleviate the need for full 2D-3D annotations.”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “Reinforcement Learning, [76, 50, 71, 45] and its variants like Meta-RL [20, 27, 54, 21, 40, 74, 17, 49] or its sister fields such as imitation learning [1, 26, 22, 89, 37, 63], commonly focus on the last part of the end-to-end active task pipeline: how to choose an action given a “state” from the wo”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, methodology): “Even more interesting is extensions to HMR that use temporal sequences [35].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, background): “to a larger framework for solving downstream active tasks. Reinforcement Learning, [76,50,71,45] and its variants like Meta-RL [20,27,54,21,40,74,17,49] or its sister ﬁelds such as imitation learning [1,26,22,89,37,63], commonly focus on the last part of the end-to-end active task pipeline: how to c”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, neutral): “We compare CLIFF with prior arts, including videobased methods [17,7,20,52] that exploit temporal information, and frame-based ones [8,25,27,23] that process each frame independently.”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, methodology): “To put our results into perspective, we also compare with four recent baselines ([3, 15, 16, 17]).”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, background): “Recently, [19, 7] forecast a person’s future 3D poses from third-person static images, which require the person to be visible.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, neutral): “method Human3.6M 3DPW MPJPE PA-MPJPE MPJPE PA-MPJPE SMPLify [6] - 82.3 - - Lassner et al. [33] - 93.9 - - HMR [26] 88.0 56.8 - 81.3 NBF [45] - 59.9 - - Pavlakos et al. [50] - 75.9 - - Kanazawa et al. [27] - 56.9 - 72.6 GraphCMR [31] - 50.1 - 70.2 Arnab et al. [4] 77.8 54.3 - 72.2 SPIN [30] - 41.1 - ”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, neutral): “Most existing works on spatialtemporal 4D reconstruction either require sufficient observations at each moment [30, 44, 45, 47, 50, 64], or are restricted to specific categories [4, 12, 21, 27, 63, 77] such as human body and faces with template models.”
PARE: Part Attention Regressor for 3D Human Body Estimation (2021, methodology): “The task is to take a single image [24, 29, 40] or video sequence [25, 27, 35] as input and to regress the parameters of a human body model such as SMPL [33] as output.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “This has led to developments in predicting motion (pose sequences) and shape directly from RGB video [42, 105, 81, 87, 22].”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, methodology): “4D Reconstruction: Most works in the area of 4D reconstruction are restricted to a ﬁxed domain by utilizing a template model [2, 5, 15, 27, 30, 63, 75], requiring a multi-view setup [33, 41, 42, 44, 45, 58, 64], or making strong assumptions about the motion e.g. rigidity or linearity…”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “Notably, recent 3D body pose estimation approaches have shown that leveraging diverse datasets can greatly improve its generalization ability [20, 21, 44, 25].”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, methodology): “Following the protocol of previous work [7, 9], we do not use its data for training but only perform evaluations on its test set.”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “For example, PHD [87], HMMR [30] can run on videos with only single person in the scene.”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “Several works [24,42,46,106] take video clips as input to exploit the temporal cues.”
Monocular, One-stage, Regression of Multiple 3D People (2020, background): “Impressive performance has been achieved using various weak supervision signals, such as 2D pose [9, 20, 25], semantic segmentation [51], geometric priors and discriminative training [20], motion dynamics [21], temporal coherence [22, 47], texture consistency [39], optimization [6] in the loop [23],”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “PennAction [64] and PoseTrack [3] are the only ground-truth 2D video datasets we use, while InstaVariety [30] and Kinetics-400 [13] are pseudo ground-truth datasets annotated using a 2D keypoint detector [12, 34].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, neutral): “9 Kanazawa [21] 56.”