Learning Joint Reconstruction of Hands and Manipulated Objects

By Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, Cordelia Schmid. 2019

658 citations recorded by the source corpus.

Estimating hand-object manipulations is essential for in- terpreting and imitating human actions. Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation. Yet, reconstructing hands and objects during manipulation is a more challeng- ing task due to significant occlusions of both the hand and object. While presenting challenges, manipulations may also simplify the problem since the physics of contact re- stricts the space of valid hand-object configurations. For example, during manipulation, the hand and object should be in contact but not interpenetrate. In this work, we regu- larize the joint reconstruction of hands and objects with ma- nipulation constraints. We present an end-to-end learnable model that exploits a novel contact loss that favors phys- ically plausible hand-object constellations. Our approach improves grasp quality metrics over baselines, using RGB images as input. To train and evaluate the model, we also propose a new large-scale synthetic dataset, ObMan, with hand-object manipulations. We demonstrate the transfer- ability of ObMan-trained models to real data.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Human-Computer Interaction

In Robotics & Control

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “VISOR is also related to efforts that capture hand-object relations, including for identifying active objects [35, 4], next active objects [33, 25], object articulation [44] and grasping [18, 39, 38, 40].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “…Our new hand state model and dataset serves as an enabling technology that lets the community deploy exciting hand-mesh reconstruction systems like [21] on YouTube videos; (3) Our system may provide a stepping stone towards important tasks like grasp analysis by showing how to use it to build a…”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “[18, 47] show that because of the domain gap between synthetic and real images, training on synthetic images only results in sub-optimal performance.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “We follow [39] and use contact area (CA), intersection volume (IV), and the ratio between contact area and intersection volume ( IV ).”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Despite the tremendous success achieved in recent years [8, 27, 28, 38, 44, 5, 16, 49, 25, 13], it remains a challenging problem due to the frequent selfocclusions and substantial depth ambiguity in 2D representations.”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, methodology): “by Zhang et al. [43] introduces a method that iteratively regresses model parameters from the heatmaps. Baek et al. [3] combine an iterative reﬁnement with a differentiable renderer and Hasson et al. [19] reconstruct hand meshes with objects they interact with. The last work by Ge et al. [15] is sim”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, methodology): “First, following [28], we build a contact prior on the hand vertices V cont that are more likely to be in contact with the target object O t and we minimize the distance between these vertices and the 3D object: V cont are computed as the vertices close to the object in at least 8% of the ground…”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “3D hand pose and shape estimation in egocentric views has started receiving attention recently [7, 22, 37, 51, 70, 72, 73].”
Reconstructing Hands in 3D with Transformers (2023, background): “The improvements in 3D hand pose estimation have also lead to progress in related problems, including joint hand pose and object reconstruction [21, 22, 54, 58] and reconstruction of two interacting hands [28, 32, 37, 38, 46, 55, 60, 61, 65].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “[10] 13.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “There are several hand-object papers that recover both 3D object and 3D hand configurations [24].”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “[19] introduces an end-to-end model to regress MANO [53] hand parameters jointly with object mesh vertices deformed from a sphere.”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “FPHA [17] (reported in [25]) 11.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, neutral): “[18] 13.”