Single-Shot Multi-person 3D Pose Estimation from Monocular RGB

By Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, Christian Theobalt. International Conference on 3D Vision, 2017

460 citations recorded by the source corpus.

We propose a new single-shot method for multi-person 3D pose estimation in general scenes from a monocular RGB camera. Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene. ORPM outputs a fixed number of maps which encode the 3D joint locations of all people in the scene. Body part associations [8] allow us to infer 3D pose for an arbitrary number of people without explicit bounding box prediction. To train our approach we introduce MuCo-3DHP, the first large scale training data set showing real images of sophisticated multi-person interactions and occlusions. We synthesize a large corpus of multi-person images by compositing images of individual people (with ground truth from mutli-view performance capture). We evaluate our method on our new challenging 3D annotated multi-person test set MuPoTs-3D where we achieve state-of-the-art performance. To further stimulate research in multi-person 3D pose estimation, we will make our new datasets, and associated code publicly available for research purposes.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dd1bfd1fa97b2a97fd82cc0103fa5c98a34b0d37

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, unknown): “Method Pavlakos [42] Zhou [73] Luo [31] Martinez [33] Fang [20] Yang [70] Pavlakos [41] Dabral [18] Sun [56] *Kanazawa [27] *Mehta [35] *Mehta [34] *Ours *Ours+”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “MuCo-3DHP [41] is synthesized from the existing MPI-INF3DHP 3D single-person pose estimation dataset [40].”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “Many of the recent deep learning methods that estimate human pose from video [15, 24, 43, 49, 44] focus on joint locations only.”
Mesh Graphormer (2021, unknown): “6M [12], MuCo-3DHP [25], UP-3D [21], COCO [23], MPII [1].”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, background): “Single-stage approaches [49, 54, 34, 74, 36, 61, 37, 69] directly estimate the 3D joint locations from the input image.”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “Recent pure bottom-up approaches to human analysis [50, 49, 58, 87, 69, 71, 62] typically predict shape represented as a coarse stick figure or bone skeleton, and can not estimate body shape or clothing.”
Deep 3D human pose estimation: A review (2021, unknown): “In fact, several works provide their new datasets, such as the Shelf and Campus datasets (Belagiannis et al., 2014), the MuCo-3DHP, and MoPoTS-3D datasets (Mehta et al., 2018), the Boxing dataset (Rhodin et al., 2019), but they have not been popularly used in other literature.”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, methodology): “Several models rely on feed-forward pose and shape prediction based on different learning architectures, on pose prediction followed by pose and shape refinement to body joints of semantic body part segmentations, or on multicamera fusion[28, 43, 44, 20, 36, 5, 18].”
Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image (2019, methodology): “The training set, MuCo-3DHP, is generated by compositing the existing MPI-INF-3DHP 3D single-person pose estimation dataset [28].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “Microsoft Common Objects in Context (COCO) Dataset [132] is the most widely used large-scale dataset.”
Monocular, One-stage, Regression of Multiple 3D People (2020, background): “[35] propose occlusion-robust pose-maps and exploit the body part association to avoid the bounding box predic-”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “MuCo-3DHP [26] is a synthesized dataset based on MPIINF-3DHP dataset [25].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “MuCo-3DHP [31] is generated by compositing the existing MPIINF-3DHP 3D [30].”
Human Motion Diffusion as a Generative Prior (2023, unknown): “MuPoTS-3D dataset [Mehta et al. 2018] includes 20 real-world multi-person sequences; CMU-Mocap [CMU [n.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “[60] introduce occlusion-robust pose-maps which enable full body pose inference for an arbitrary number of people even under partial occlusions.”