Monocular Total Capture: Posing Face, Body, and Hands in the Wild

By Donglai Xiang, Hanbyul Joo, Yaser Sheikh. 2018

363 citations recorded by the source corpus.

We present the first method to capture the 3D total motion of a target person from a monocular view input. Given an image or a monocular video, our method reconstructs the motion from body, face, and fingers represented by a 3D deformable mesh model. We use an efficient representation called 3D Part Orientation Fields (POFs), to encode the 3D orientations of all body parts in the common 2D image space. POFs are predicted by a Fully Convolutional Network, along with the joint confidence maps. To train our network, we collect a new 3D human motion dataset capturing diverse total body motion of 40 subjects in a multiview system. We leverage a 3D deformable human model to reconstruct total body pose from the CNN outputs with the aid of the pose and shape prior in the model. We also present a texture-based tracking method to obtain temporally coherent motion capture output. We perform thorough quantitative evaluations including comparison with the existing body-specific and hand-specific methods, and performance analysis on camera viewpoint and human pose changes. Finally, we demonstrate the results of our total body motion capture on various challenging in-the-wild videos.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Most existing approaches thus limit themselves to a single object category, such as the human body [23, 91, 32] or face [12].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, methodology): “Some methods [5, 32, 9] estimated the low-dimensional parameters for a 3D deformable hand model [19] to fit the RGB inputs in order to retrieve the 3D canonical hand poses.”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, methodology): “The follow-up work [36] enables predicting the model parameters in the wild by fitting the model into predicted joint confidence and orientation maps.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “1 2019 [259] Adam Image - - - - - - - - - - - - - - - 58.”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Fitting-based approaches [32, 52, 19] fit such parametric models to 2D keypoint detections to optimize 3D hand parameters.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “, avoiding foreshortening, we find that state-of-the-art 3D methods [29, 62, 71] fail on our data due to extensive occlusions from the wearer’s hands.”
Conv-MPN: Convolutional Message Passing Neural Network for Structured Outdoor Architecture Reconstruction (2019, background): “Convolutional neural networks have shown to be effective in solving human pose estimation [22, 31, 29] and hand tracking [34, 38, 30].”
Reconstructing Hands in 3D with Transformers (2023, methodology): “For training data, we use multiple available sources of data with hand annotations, including both studio/controlled datasets with 3D ground truth [6, 19, 40, 56, 63, 64], and in-the-wild datasets annotated with 2D keypoint locations [15, 25, 52].”
Whole-Body Human Pose Estimation in the Wild (2020, methodology): “Recent works [24,47,64] also explore the task of monocular 3D whole-body capture.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “To estimate joint rotations, [47, 29] fitted a generic hand model to the predictions via an iterative optimization based approach, which is not time-efficient and requires hand-crafted energy functionals.”
Learnable Triangulation of Human Pose (2019, methodology): “The CMU panoptic is a new multi-camera dataset maintained by the Carnegie Mellon University [5, 20, 15]. cameras acquired via triangulation using all camera views.”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “Body Mesh Recovery Approaches based on regressing MANO parameters described in the previous subsection originate from works on body mesh recovery [3, 28, 22, 48, 42, 50, 41, 39, 21, 55, 25].”
On Self-Contact and Human Pose (2021, methodology): “Recovering human meshes from images typically involves either learning a regressor from pixels to 3D pose and shape [20, 23], or fitting a 3D model to image features using an optimization method [4, 34, 45, 46].”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “which constraints meshes to the space of humans. Most of current model based approaches optimize the pose and shape of SMPL [27] to match image features, which are extracted with bottom-up predictors [58,7,8,12,53]. Alternative methods based on GHUM[54] 4 BL. Bhatnagar et al. also exist[57]. The mos”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “Monocular Total Capture (MTC) [55] is based on the Adam model [19].”
Articulation-Aware Canonical Surface Mapping (2020, background): “movement of parts, and a plethora of fitting based [4, 15] or prediction based [14, 36, 43] methods have been proposed to tackle this.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “More recent approaches employ expressive human models with faces and fingers [65,50].”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “[100] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.”