Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

By Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, Michael J. Black. IEEE International Conference on Computer Vision, 2019

404 citations recorded by the source corpus.

To understand and analyze human behavior, we need to capture humans moving in, and interacting with, the world. Most existing methods perform 3D human pose estimation without explicitly considering the scene. We observe however that the world constrains the body and vice-versa. To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene. Our key contribution is to exploit static 3D scene structure to better estimate human pose from monocular images. The method enforces Proximal Relationships with Object eXclusion and is called PROX. To test this, we collect a new dataset composed of 12 different 3D scenes and RGB sequences of 20 subjects moving in and interacting with the scenes. We represent human pose using the 3D human body model SMPL-X and extend SMPLify-X to estimate body pose using scene constraints. We make use of the 3D scene information by formulating two main constraints. The inter-penetration constraint penalizes intersection between the body model and the surrounding 3D scene. The contact constraint encourages specific parts of the body to be in contact with scene surfaces if they are close enough in distance and orientation. For quantitative evaluation we capture a separate dataset with 180 RGB frames in which the ground-truth body pose is estimated using a motion capture system. We show quantitatively that introducing scene constraints significantly reduces 3D joint error and vertex error. Our code and data are available for research at https://prox.is.tue.mpg.de.

View Paper: https://www.semanticscholar.org/paper/c6db4ab2622533d161cde8838053f74c5c4eb3b6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “While recent work has also demonstrated impressive results in estimating 3D structures, particularly human bodies, the focus is often on bodies [30,37] and objects [11,17] imaged in isolation or in controlled lab conditions [23,56].”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “The PROX-D baseline uses the fittings provided with the PROX dataset, which are on the known gendered SMPL-X body model and use face/hand 2D keypoints for fitting.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “PROX [33] is easier to scale as it uses a single Kinect based capture setup (although, scene needs to be pre-scanned) but this reduces the overall quality.”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, unknown): “Recent works [5, 29, 32, 35] do explore human-object relations for more accurate 3D reconstruction, but they only deals with static objects with no part motion.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Furthermore, in contrast to previous works, e.g. [5,78,79], which can only evaluate body-object interactions after obtaining the body and the object meshes, when using GF as the representation in hand-object reconstruction from images, we model the hand, the object, and the contact area by the…”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Collision penalization is implemented either with approximate shape primitives [3, 34, 48] or triangle meshes [2, 16, 17, 19, 23, 33, 50, 60, 65].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “For example, [13] requires the input of full 3D reconstructed scene mesh to impose geometry constraints.”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, background): “Most research has focused on hand pose estimation [17,53,57,59], realistic hand and body reconstruction [24, 25, 61, 65], and robotic grasp prediction for anthropomorphic hands [7, 36].”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, methodology): “[23] use RGB-D and 3D scene constraints to capture 20 humans as SMPL-X [46] meshes interacting with 12 static 3D scenes, but do not capture object manipulation.”
Deep 3D human pose estimation: A review (2021, unknown): “For example, Hassan et al. (2019) try to exploit the context information of a 3D scene by using the constraint that the human body model and the scene model cannot share the same 3D space.”
On Self-Contact and Human Pose (2021, background): “Physical constraints can come from the ground plane [44, 49], an object [13, 21, 22], or contextual scene information [11, 47].”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “Furthermore, DSGs provide a framework to model plausible interactions between agents and scenes (Zhang et al. 2019b; Hassan et al. 2019; Pirk et al. 2017; Monszpart et al. 2019).”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, background): “Furthermore, DSGs provide a framework to model plausible interactions between agents and scenes [31, 70, 82, 115].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[163] further extended SMPLify-X to PROX – a method enforcing Proximal Relationships with Object eXclusion by adding 3D environmental constraints.”