H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

By Bugra Tekin, Federica Bogo, Marc Pollefeys. 2019

287 citations recorded by the source corpus.

We present a unified framework for understanding 3D hand and object interactions in raw image sequences from egocentric RGB cameras. Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end on single images. We further merge and propagate information in the temporal domain to infer interactions between hand and object trajectories and recognize actions. The complete model takes as input a sequence of frames and outputs per-frame 3D hand and object pose predictions along with the estimates of object and action categories for the entire sequence. We demonstrate state-of-the-art performance of our algorithm even in comparison to the approaches that work on depth data and ground-truth annotations.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “In this area, our data and approach sit between image-based hand detection datasets like [1, 25, 5, 14, 28] and efforts at understanding contact with richer annotation but requiring more specialized devices or more constrained environments such as [7, 35, 30, 15, 36].”
Detecting Hands and Recognizing Physical Contact in the Wild (2020, methodology): “[30] propose to model hand-object interactions by jointly estimating the 3-D poses for hand and objects.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “We ﬁrst report the pose estimation accuracy of our single-frame hand-object reconstruction model and compare it against the state of the art [10, 47].”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “[57] designs a neural network to jointly predict 3D hand pose and 3D object bounding boxes with a focus on egocentric scenarios.”
Human Hands as Probes for Interactive Object Understanding (2021, background): “Several works have sought to build a data-driven understanding of human hands and how they manipulate objects from RGB images [63], RGB-D images [51], egocentric data [31], videos [17] and other sensors [2, 58].”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, result): “Note that this is a commonly used setting in previous works (e.g. [88,90,91]).”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “[66] infer 3D control points for both the hand and the object in videos, using a temporal model to propagate information across time.”
Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, methodology): “Modeling and analysing hand-object interaction is an active ﬁeld of study with two main paradigms: joint estimating hand-object poses simultaneously during interaction [16, 47, 19, 13, 35, 39, 21, 29] and studying from multi-modal hand-object interaction representations [17, 36, 2, 4, 46, 45, 14].”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, methodology): “This dataset is used by [33, 66] to learn to estimate 3D hand and object poses, but suffers from noisy poses and significant inter-penetrations (see Sec.”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Other works proposed similar methods for monocular pose estimation [25, 18, 27].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “Traditionally, these two problems have been addressed separately, yet in many critical applications, we need both capabilities working together [36, 6, 13].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “These include attention mechanisms from gaze and motor attention [47, 38, 42], active object detection [20, 7, 15, 70], and hands in contact [66, 60, 33].”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, background): “This investigation is complementary to recent work on image-based estimation of object geometry [20,68], object pose [18,60], and hand pose [23,53,57,65,69].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “Joint reconstruction of hands and objects has recently received increased attention [8, 13, 16, 18, 19, 59].”