Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image

By Denis Tomè, Chris Russell, Lourdes de Agapito. 2017

532 citations recorded by the source corpus.

We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search for better 2D locations. The entire process is trained end-to-end, is extremely efficient and obtains state-of-the-art results on Human3.6M outperforming previous approaches both on 2D and 3D errors.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, methodology): “[6] proposed to perform 2D joint estimation and 3D pose reconstruction jointly to improve both tasks.”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “Recently, deep architectures have been used to learn precise 3D representations from RGB images [60, 50, 30, 49, 31, 39], thanks to the availability of high quality data [21], and are now able to surpass depthsensors [32].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “The second protocol [5, 44], P2, uses the same train/test set, but is tested only on the frontal camera (camera 3) and reports reconstruction error.”
Compositional Human Pose Regression (2017, methodology): “Some methods (Pavlakos et al., 2016; Tome et al., 2017) firstly train the deep network model on 2D data and fine-tune the model on 3D data.”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, methodology): “Most previous work [46, 2, 32] in 3D human pose and hand pose estimation based on single image attempt to lift the set of 2D heatmaps into 3D space dirctly, while a key issue for this strategy is how to distinguish between the multiple 3D poses inferred from a single 2D skeleton.”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, background): “The research can be categorized into sparse 2D parsing (2D skeleton estimation) [6, 39], sparse 3D parsing (3D skeleton estimation) [33, 44, 64, 48, 50, 35, 61], dense 2D parsing [17] and dense 3D parsing (shape and pose estimation).”
Integral Human Pose Regression (2017, methodology): “For evaluation, many previous works [8,46,32,54,25,31,37,51,41,4,53,44,56] use the mean per joint position error ( MPJPE ).”
Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image (2019, unknown): “Tome [46] 65.”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “, [51, 64, 48, 25]) also use in-the-wild images with 2D keypoints for supervision, most of the improvement for our approach comes from augmenting training with ordinal depth relations for these examples.”
Exploiting Temporal Information for 3D Human Pose Estimation (2017, background): “[40] also extended a 2D pose estimator called Convolutional Pose Machine (CPM) [17] to make it predict 3D pose.”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, background): “Walking and upright poses are prime examples of this, and state-of-the-art algorithms [18, 27, 20, 15, 17] now deliver impressive real-time results in uncontrolled environments.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, background): “Finally, some recent approaches treat 2D and 3D pose estimation jointly or iteratively [45], [46], [47], [48], [49].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “Recent pure bottom-up approaches to human analysis [50, 49, 58, 87, 69, 71, 62] typically predict shape represented as a coarse stick figure or bone skeleton, and can not estimate body shape or clothing.”
Video Based Reconstruction of 3D People Models (2018, background): “Recent works leverage 2D annotations to train networks for the task of 3D pose estimation [42, 53, 84, 65, 68, 56].”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, methodology): “A standard approach to address the domain difference between 3D human pose estimation datasets and images in the wild is to split the task into two separate subtasks [34, 26, 5, 3, 30].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Some approaches jointly reason about 2D and 3D pose with multi-stage belief maps [62].”
Benchmarking and Error Diagnosis in Multi-instance Pose Estimation (2017, background): “…for improvement are possible: (i) collecting 3D annotations [7] for the humans in COCO and learning to directly regress 3D pose end-to-end [30]; (ii) modeling the manifold of human poses [3, 6, 35] and learning how to jointly predict the 3D pose of a person along with its 2D skeleton [41].”
Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors (2017, methodology): “We use the standard evaluation protocol as followed by [16, 19, 28, 29, 30] where subjects S1, S5, S6, S7, S8 are used for training and Subjects S9, S11 provide the test sequences.”
Deep learning tools for the measurement of animal behavior in neuroscience (2019, background): “Yet currently, the highest accuracy is achieved by using multiple 2D views to reconstruct a 3D estimate (Figure 1b; (20, 35)), but other ways of “lifting" 2D into 3D are being actively explored (20, 36, 37).”