Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild

By Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, Angjoo Kanazawa. 2020

181 citations recorded by the source corpus.

We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene, all from a single image in-the-wild captured in an uncontrolled environment. Notably, our method runs on datasets without any scene- or object-level 3D supervision. Our key insight is that considering humans and objects jointly gives rise to "3D common sense" constraints that can be used to resolve ambiguity. In particular, we introduce a scale loss that learns the distribution of object size from data; an occlusion-aware silhouette re-projection loss to optimize object pose; and a human-object interaction loss to capture the spatial layout of objects with which humans interact. We empirically validate that our constraints dramatically reduce the space of likely 3D spatial configurations. We demonstrate our approach on challenging, in-the-wild images of humans interacting with large objects (such as bicycles, motorcycles, and surfboards) and handheld objects (such as laptops, tennis rackets, and skateboards). We quantify the ability of our approach to recover human-object arrangements and outline remaining challenges in this relatively domain. The project webpage can be found at this https URL.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “[57] requires manually labeling human-object mesh vertices for fine-grained interaction pairs and is only applied to 8 object categories in COCO [25].”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, background): “Recent works [5, 29, 32, 35] do explore human-object relations for more accurate 3D reconstruction, but they only deals with static objects with no part motion.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, result): “We compare our method with PHOSA [9] and show that our approach generates noticeably better quality results.”
Boundary IoU: Improving Object-Centric Image Segmentation Evaluation (2021, background): “Note that mask boundary quality is an essential aspect of image segmentation, as various downstream applications directly benefit from more precise object segmentations [39, 33, 34].”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “However, while [65] fits objects among the COCO categories, our target everyday objects are often not present among the COCO classes.”
MonoScene: Monocular 3D Semantic Scene Completion (2021, background): “For multiple objects, a common practice is to couple reconstruction with object detection [27,28,34,35,78].”