Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

By Alin-Ionut Popa, Mihai Zanfir, Cristian Sminchisescu. 2017

154 citations recorded by the source corpus.

We propose a deep multitask architecture for fully automatic 2d and 3d human sensing (DMHS), including recognition and reconstruction, in monocular images. The system computes the figure-ground segmentation, semantically identifies the human body parts at pixel level, and estimates the 2d and 3d pose of the person. The model supports the joint training of all components by means of multi-task losses where early processing stages recursively feed into advanced ones for increasingly complex calculations, accuracy and robustness. The design allows us to tie a complete training protocol, by taking advantage of multiple datasets that would otherwise restrictively cover only some of the model components: complex 2d image data with no body part labeling and without associated 3d ground truth, or complex 3d data with limited 2d background variability. In detailed experiments based on several challenging 2d and 3d datasets (LSP, HumanEva, Human3.6M), we evaluate the sub-structures of the model, the effect of various types of training data in the multitask loss, and demonstrate that state-of-the-art results can be achieved at all processing levels. We also show that in the wild our monocular RGB architecture is perceptually competitive to a state-of-the art (commercial) Kinect system based on RGB-D data.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Human part segmentation masks have been provided in a number of datasets [48, 6, 13]; these can be understood as providing a coarsened version of image-to-surface correspondence, where rather than continuous coordinates one predicts discretized part labels [35].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “Recently, deep architectures have been used to learn precise 3D representations from RGB images [60, 50, 30, 49, 31, 39], thanks to the availability of high quality data [21], and are now able to surpass depthsensors [32].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Another approach is to train a network with separate 2D and 3D losses for the diﬀer-ent data sources [46, 68, 56, 65, 31].”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, methodology): “Walking and upright poses are prime examples of this, and state-of-the-art algorithms [18, 27, 20, 15, 17] now deliver impressive real-time results in uncontrolled environments.”
Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera (2018, methodology): “For every example we show from left to right: a cropped image of the person to be estimated, our results, method [3], method [4] and method [2].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “Recent pure bottom-up approaches to human analysis [50, 49, 58, 87, 69, 71, 62] typically predict shape represented as a coarse stick figure or bone skeleton, and can not estimate body shape or clothing.”
Deep 3D human pose estimation: A review (2021, unknown): “They infer the 2D pose, 3D pose, and semantic body parts of multiple people by the deep multitask human sensing network (DMHS, Popa et al. (2017)).”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, background): “Many image and video-based pose and shape estimation methods have also been proposed[33, 27, 25, 40, 22, 1, 12, 26, 34, 23].”
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera (2018, background): “Recently 2D annotations have been leveraged to train networks for the task of 3D pose estimation [21, 28, 36, 38, 50].”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “Data augmentation using natural images with 2D keypoint annotations is a standard practice [48, 26, 36, 64, 44], but here we also consider the effect of our ordinal depth supervision.”
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints (2018, methodology): “DMHS [23] is a state-of-the-art feedforward multi-task deep neural network for human sensing that provides, for a given image I ∈ RW×H×3, the following estimates: the 2d and 3d joints of a single person as well as the semantic body parts at pixel level.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “The feed forward process includes semantic segmentation of body parts and 3D pose estimates based on DMHS (Popa et al., 2017).”
BodyNet: Volumetric Inference of 3D Human Body Shapes (2018, unknown): “Unlike [51], 3D pose becomes an auxiliary task for our final 3D shape task.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, background): “lothing jointly from scans [35]. These approaches are typically slow, and are susceptible to local-minima. In contrast, deep learning based models predict body model parameters in a feed-forward pass [16,43,39] and use bottom-up 2D features for self-supervision [21,23,24,32,50,60] during training. T”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “Other approaches formulate the problem in a multi-task setup to jointly estimate both 2D keypoint locations and 3D pose [27, 28, 52–54].”
A Simple Yet Effective Baseline for 3d Human Pose Estimation (2017, background): “Modern systems [12, 26, 34, 39] typically approach this problem from a supervised learning perspective, letting the system infer which image features are most discriminative for depth estimation.”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, background): “[20] consider 3D human pose estimation as a multi-task learning of 2D and depth regression with different data.”
Video Based Reconstruction of 3D People Models (2018, background): “Recent works leverage 2D annotations to train networks for the task of 3D pose estimation [42, 53, 84, 65, 68, 56].”
Learning to Estimate 3D Hand Pose from Single RGB Images (2017, background): “Recently, there have been several approaches that apply deep learning for lifting 2D keypoints to 3D pose for human body pose estimation [26, 11, 16].”