Reconstructing Hand-Object Interactions in the Wild

By Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, Jitendra Malik. 2020

192 citations recorded by the source corpus.

We study the problem of understanding hand-object interactions from 2D images in the wild. This requires reconstructing both the hand and the object in 3D, which is challenging because of the mutual occlusion between the hand and the object. In this paper we make two main contributions: (1) a novel reconstruction technique, RHO (Reconstructing Hands and Objects), which reconstructs 3D models of both the hand and the object leveraging the 2D image cues and 3D contact priors; (2) a dataset MOW (Manipulating Objects in the Wild) of 500 examples of hand-object interaction images that have been "3Dfied" with the help of the RHO technique. Overall our dataset contains 121 distinct object categories, with a much greater diversity of manipulation actions, than in previous datasets.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “[6] used segmentation masks of hands and interacting objects to compute 2D projection loss in order to optimize the 3D mesh reconstructions of hand-object pairs.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “Human-object datasets: Several datasets [1, 7, 46, 61, 70, 73] contain images of human-object interaction, but here we focus on large-scale data [3, 19, 22, 25, 27, 55, 89] that facilitates machine learning.”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “Understanding the interactions between a camera wearer and the surrounding objects is a fundamental problem in First Person Vision (FPV) [19, 87, 57, 33, 20, 73, 12, 4, 5, 13].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “…taxonomy [15], hand keypoint and pose estimation [17], understanding gestures [18], detecting hands, their states and objects of interaction [55, 56], 3D reconstruction of the hand and the object of interaction [4, 21], or even estimating forces being applied by the hand onto the object [13].”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Concurrent work of [8] extends the optimization-based body-object reconstruction method PHOSA [65] to perform hand-object fitting.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Approaches also track human poses, especially hands [14, 18, 65, 66, 101, 107, 127].”