VNect

By Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, Christian Theobalt. 2017

874 citations recorded by the source corpus.

We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, methodology): “VNect [28] fits a rigged skeleton model over time to estimated 2D and 3D joint locations.”
Compositional Human Pose Regression (2017, methodology): “, 2018), fitting a 3D model via optimization (Zhou et al., 2016c; Bogo et al., 2016; Zhou et al., 2017b; Mehta et al., 2017; Kanazawa et al., 2017), searching over a database (Chen and Ramanan, 2016), or generating multiple hypotheses (Jahangiri and Yuille, 2017).”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “Recent pure bottom-up approaches to human analysis [50, 49, 58, 87, 69, 71, 62] typically predict shape represented as a coarse stick figure or bone skeleton, and can not estimate body shape or clothing.”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, result): “In Figure 4 we have collected a sample of 3D pose output for our approach, focusing on MPI-INF-3DHP, since it is the main dataset that we evaluate without touching the training data.”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Moreover, to deal with the incoherent and jittery predictions, some work [17, 39, 30] turned to exploit the temporal information across sequences.”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, background): “Sit Down Smoke Photo Wait Walk Dog Pair All Mehta [54] 62.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, background): “Now, the 3D detector takes the feature maps F and the heat maps H , and estimates 3D hand joint positions in the form of location maps L, similar to [20].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “There are also approaches that use a hybrid framework by using a deep learning framework to produce an intermediate representation such as 2D and depth heat maps and fitting a skeletal model on these outputs to reconstruct joint angles [31, 55, 30].”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, methodology): “Since VNect can only estimate a single person, we design an additional baseline for 3D multi-person human pose estimation in the world coordinate.”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, methodology): “In addition, we evaluate VNect [34] on images cropped with the ground truth bounding box around the subject.”
Video Based Reconstruction of 3D People Models (2018, background): “Recent works leverage 2D annotations to train networks for the task of 3D pose estimation [42, 53, 84, 65, 68, 56].”
3D Human Pose Estimation in the Wild by Adversarial Learning (2018, methodology): “[27] built a real-time 3D pose estimation solution with kinematic skeleton fitting.”
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (2018, methodology): “More related to our work, a number of recent CNN-based techniques have been developed for 3D human keypoint detection from monocular RGB images, which use various architectures, supervised objectives, and 3D structural priors to directly infer a predefined set of 3D joint locations [36, 34, 8, 35, 1”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, neutral): “Deep learning based approaches [70, 35, 32, 57] have also succeeded in directly regressing images to 3D joint locations with the help of large-scale MoCap datasets [15].”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, background): “A growing number of computer vision and machine learning applications require 2D human pose estimation as an input for their systems [56], [57], [58], [59], [60], [61], [62].”
Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “[167], [168] and depth data [169], [170], [171], [172], [173], [174].”
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies (2018, background): “A recent collection of approaches aims to re-construct 3D skeletons directly from monocular images, either by ﬁtting 2D keypoint detections with a prior on human pose [60, 8] or getting even closer to direct regression meth-ods [61, 34, 48].”