Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach

By Xingyi Zhou, Qi-Xing Huang, Xiao Sun, Xiangyang Xue, Yichen Wei. 2017

609 citations recorded by the source corpus.

In this paper, we study the task of 3D human pose estimation in the wild. This task is challenging due to lack of training data, as existing datasets are either in the wild images with 2D pose or in the lab images with 3D pose.,, We propose a weakly-supervised transfer learning method that uses mixed 2D and 3D labels in a unified deep neutral network that presents two-stage cascaded structure. Our network augments a state-of-the-art 2D pose estimation sub-network with a 3D depth regression sub-network. Unlike previous two stage approaches that train the two sub-networks sequentially and separately, our training is end-to-end and fully exploits the correlation between the 2D pose and depth estimation sub-tasks. The deep features are better learnt through shared representations. In doing so, the 3D pose labels in controlled lab environments are transferred to in the wild images. In addition, we introduce a 3D geometric constraint to regularize the 3D pose prediction, which is effective in the absence of ground truth depth labels. Our method achieves competitive results on both 2D and 3D benchmarks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, methodology): “Method [10] [38] 3D Pose Net 3D Pose Net with FB+”
Compositional Human Pose Regression (2017, background): “ed after using MPII data. For example, joint error is reduced from 102:2 to 64:2. This improvement should originate from the better learnt feature from the abundant 2D data. See the contemporary work [53] for more discussions. Note that adding 2D data in this work is simple and not considered as a m”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “[68] (64.9mm), Mehta et al .”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, methodology): “To achieve better performance for in-the-wild images, weakly-supervised methods [68, 44, 18] have been proposed to learn from images without annotations.”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “Alternatively, the 3D pose can be predicted from a given set of 2D keypoints by simply predicting their depth [60].”
Deep 3D human pose estimation: A review (2021, neutral): “…input images to 3D body joint positions can be categorized into two classes: detection-based methods (Pavlakos et al., 2017a; Luvizon et al., 2018) and regression-based methods (Li and Chan, 2014; Zhou et al., 2016a; Sun et al., 2017; Tekin et al., 2017; Zhou et al., 2017; Luvizon et al., 2019).”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “Direct estimation methods predict 3D human pose directly from images, in the form of direct coordinate regression [46, 55, 56], voxel [42, 32, 66] or depth map [73].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “In those cases, a weak-perspective camera model is often applied to approximate the 3D-to-2D transformation for keypoint estimation [51,26], by changing Eq.”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Single-stage approaches [51, 56, 36, 76, 38, 63, 39, 71] directly estimate the 3D joint locations from the input image.”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Many existing works [3, 12, 17, 29, 54, 53, 15, 14] rely on effective 2D pose estimation frameworks to first localize the 2D keypoints on the image plane, and then lift 3D poses from the estimated 2D joint positions.”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, background): “There is rich recent literature on 3D pose estimation in the form of a simplistic body skeleton, e.g., [3, 19, 22, 24, 25, 29, 30, 34, 35, 38, 40, 41, 42, 50, 51].”
Learning to Estimate 3D Human Pose and Shape from a Single Color Image (2018, background): “Many recent works follow the end-to-end paradigm [48, 40, 42, 46, 55], using images as input to predict 3D joint locations [23, 45, 34, 28], regress 3D heatmaps [31], or classify the image in a particular pose class [39, 40].”
Integral Human Pose Regression (2017, methodology): “Regression methods are widely used for 3D pose estimation [38, 45, 46, 25, 27, 24, 30, 39].”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, methodology): “, avoiding foreshortening, we find that state-of-the-art 3D methods [29, 62, 71] fail on our data due to extensive occlusions from the wearer’s hands.”
Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation (2017, methodology): “(Zhou et al. 2017) mixed 2D and 3D data and trained an unified network with two-stage cascaded structure.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Other works regress poses directly from pixels [46, 55, 56, 68, 71, 77, 78, 96], which generally relies on having a good match between training and testing.”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, background): “However, instead of estimating 3D human pose from monocular images, videos can provide temporal information to improve accuracy and robustness [49, 10, 32, 8, 2, 44, 38].”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “2 mm obtained with LCR-Net+) and outperform all previously published methods, including very recent work, despite the fact that (a) we also perform localization, in contrast to most methods such as [8], [43], [51] that assume a bounding box annotation of the human and (b) we propose an end-to-end ar”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “, [51, 64, 48, 25]) also use in-the-wild images with 2D keypoints for supervision, most of the improvement for our approach comes from augmenting training with ordinal depth relations for these examples.”