Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints

By Andrei Zanfir, Elisabeta Marinoiu, Cristian Sminchisescu. 2018

313 citations recorded by the source corpus.

Human sensing has greatly benefited from recent advances in deep learning, parametric human modeling, and large scale 2d and 3d datasets. However, existing 3d models make strong assumptions about the scene, considering either a single person per image, full views of the person, a simple background or many cameras. In this paper, we leverage state-of-the-art deep multi-task neural networks and parametric human and scene modeling, towards a fully automatic monocular visual sensing system for multiple interacting people, which (i) infers the 2d and 3d pose and shape of multiple people from a single image, relying on detailed semantic representations at both model and image level, to guide a combined optimization with feedforward and feedback components, (ii) automatically integrates scene constraints including ground plane support and simultaneous volume occupancy by multiple people, and (iii) extends the single image model to video by optimally solving the temporal person assignment problem and imposing coherent temporal pose and motion reconstructions while preserving image alignment fidelity. We perform experiments on both single and multi-person datasets, and systematically evaluate each component of the model, showing improved performance and extensive multiple human sensing capability. We also apply our method to images with multiple people, severe occlusions and diverse backgrounds captured in challenging natural scenes, and obtain results of good perceptual quality.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, methodology): “Recently, they have also been successfully applied to 3D pose estimation [5], [6], [8], [10], [11], [12], [13].”
Monocular, One-stage, Regression of Multiple 3D People (2020, methodology): “Existing approaches [15, 22, 52, 53] follow a multi-”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “…et al., 2017c; Rhodin et al., 2018a) (3) Other model-based : probabilistic model (Tome et al., 2017) 3D Multiple bottom-up (Mehta et al., 2017b), top-down (Rogez et al., 2017), SMPL-based (Zanﬁr et al., 2018), real-time (Mehta et al., 2019) cations in the 2D surface, then extend them to 3D space.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “Meshes are common to represent humans and have been used for estimating pose, shape [38, 40, 41, 52, 76, 86] and clothing [3, 2, 11] from images, but topologies and detail are restricted by an underlying model like SMPL [47] or SMPL + Garments meshes (ClothCap [57], TailorNet [54] and GarNet [30],…”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, background): “[73] establish contact constraints between the feet and an estimated ground plane.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, background): “which results in miss-alignments between reconstructions and input pixels. Hybrid methods mitigate these problems by rening feed-forward predictions with optimization at training[57] and/or test time[59], and by predicting displacements on top of SMPL, demonstrating capture of ne details and even cl”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, methodology): “[104] Andrei Zanfir, Elisabeta Marinoiu, and Cristian Sminchisescu.”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, methodology): “Several models rely on feed-forward pose and shape prediction based on different learning architectures, on pose prediction followed by pose and shape refinement to body joints of semantic body part segmentations, or on multicamera fusion[28, 43, 44, 20, 36, 5, 18].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Of 1 A second approach [67] was published in the review period. the two exceptions, the MARCOnI dataset [12] features 5 sequences but contains only 2 persons simultaneously, and there are no close interactions.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Tasks like placement of humans into static scenes [34, 49, 90], motion prediction [15,32] or human pose reconstruction [16,33,77,86, 89] under scene constrains, or learning priors for human-object interactions [66], have been investigated extensively in recent years.”
Learning 3D Human Dynamics From Video (2018, background): “Finally, we discuss how to transfer this knowledge to static images such that one can hallucinate short-term human dynamics from a static image.”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, methodology): “Based on this paradigm, different updates have been investigated to incorporate information such as 2D/3D body joints [5, 68], silhouettes [29, 16], part segmentation [63] in the fitting procedure.”
On Self-Contact and Human Pose (2021, methodology): “[49] use a volume occupancy exclusion loss to prevent penetration.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “More related to our work are methods that recover 3D pose and shape of multiple people [67,68,27].”
STAR: Sparse Trained Articulated Human Body Regressor (2020, background): “This makes it useful to reason about 3D human body pose and shape given sparse measurements, such as IMU accelerations [11,12,26], sparse mocap markers [22,25] or 2D key points in images and videos [3,14,15,16,28,33,36,38].”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “(2020) for a broader review, it is worth mentioning that related work includes optimization-based approaches, which fit a 3D mesh to 2D image keypoints (Bogo et al., 2016; Kolotouros et al., 2019a; Lassner et al., 2017; Yang and Carlone, 2020; Zanfir et al., 2018), and learning-based methods, which ”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “[280] added semantic segmentation to the 3D multi-person HPE module with scene constraints.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for humans: Since the introduction of the (mesh-based) SMPL human model [41] there have been a growing number of papers leveraging it to reconstructing shape and pose from point clouds, depth data and images [28, 32, 33, 47, 69, 79].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Starting with a feed-forward 3D prediction, semantic segmentation, keypoints and scene constraints are integrated top-down in order to predict the pose and shape of multiple people [81].”