Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense

By Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, Song-Chun Zhu. 2019

131 citations recorded by the source corpus.

We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation. The intuition behind is to leverage the coupled nature of these two tasks to improve the granularity and performance of scene understanding. We propose to exploit two critical and essential connections between these two tasks: (i) human-object interaction (HOI) to model the fine-grained relations between agents and objects in the scene, and (ii) physical commonsense to model the physical plausibility of the reconstructed scene. The optimal configuration of the 3D scene, represented by a parse graph, is inferred using Markov chain Monte Carlo (MCMC), which efficiently traverses through the non-differentiable joint solution space. Experimental results demonstrate that the proposed algorithm significantly improves the performance of the two tasks on three datasets, showing an improved generalization ability.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “However, others propose to create 3D scene graphs that encode 3D relationships between the components of a scene [2, 12, 20, 51, 66].”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “Such a setting potentially beneﬁts future study on 3D holistic scene understanding [10,25], as well as action understanding and prediction [48,42].”
Deep 3D human pose estimation: A review (2021, unknown): “Another approach to encoding geometry representation is to encode the 2D pose to a latent one of the 3D pose with a representation consistency constraint (Chen et al., 2019a).”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Tasks like placement of humans into static scenes [34, 49, 90], motion prediction [15,32] or human pose reconstruction [16,33,77,86, 89] under scene constrains, or learning priors for humanobject interactions [66], have been investigated extensively in recent years.”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, methodology): “modeled support relationships as edges in a human-centric scene graphical model, inferred the relationships by minimizing supporting energies among objects and the room layout [36], and enforced physical stability and plausibility by penalizing the intersections among reconstructed 3D objects and ro”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “Thus, applications such as action recognition [95, 96] or holistic dynamic indoor scene understanding [19] require accurate perception of 3D human pose, shape, motion, contacts, and interaction.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “Most related to our work is [9], who develop an approach that recovers a parse graph that represents the 3D human pose, 3D object, and scene layout from a single image.”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, background): “In [3], the authors build a graph based on human-object relations and propose a holistic scene parsing and reconstruction pipeline.”