Grasping Field: Learning Implicit Representations for Human Grasps

By Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J. Black, Krikamol Muandet, Siyu Tang. 2020

255 citations recorded by the source corpus.

Robotic grasping of house-hold objects has made remarkable progress in recent years. Yet, human grasps are still difficult to synthesize realistically. There are several key reasons: (1) the human hand has many degrees of freedom (more than robotic manipulators); (2) the synthesized hand should conform to the surface of the object; and (3) it should interact with the object in a semantically and physically plausible manner. To make progress in this direction, we draw inspiration from the recent progress on learning-based implicit representations for 3D object reconstruction. Specifically, we propose an expressive representation for human grasp modelling that is efficient and easy to integrate with deep neural networks. Our insight is that every point in a three-dimensional space can be characterized by the signed distances to the surface of the hand and the object, respectively. Consequently, the hand, the object, and the contact area can be represented by implicit surfaces in a common space, in which the proximity between the hand and the object can be modelled explicitly. We name this 3D to 2D mapping as Grasping Field, parameterize it with a deep neural network, and learn it from data. We demonstrate that the proposed grasping field is an effective and expressive representation for human grasp generation. Specifically, our generative model is able to synthesize high-quality human grasps, given only on a 3D object point cloud. The extensive experiments demonstrate that our generative model compares favorably with a strong baseline and approaches the level of natural human grasps. Furthermore, based on the grasping field representation, we propose a deep network for the challenging task of 3D hand-object interaction reconstruction from a single RGB image. Our method improves the physical plausibility of the hand-object contact reconstruction and achieves comparable performance for 3D hand reconstruction compared to state-of-the-art methods. Our model and code are available for research purpose at https://github.com/korrawe/grasping_field.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “A line of works [2,7,21,23,67] have also attempted to generate hand pose conditioned on objects.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Other methods focus on the grasp synthesis [11, 25, 58], or contact modeling [5, 6] given a 3D object.”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, unknown): “Karunratanakul [12] deﬁnes a grasping ﬁeld to jointly reconstructs the human hand and interacted object.”
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, background): “[26] infer a “grasping” distance field and then fit MANO to it.”
Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, methodology): “To generate a realistic grasp, Karun-ratanakul et al. [24] proposed an implicit representation for modeling the joint distribution of hand-object shape.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Dynamic human object interactions Recently, there has been a strong push on modeling hand-object interactions based on 3D [42,72], 2.5D [13,14] and 2D [22,24,28,35,83] Dataset RGBD Hum.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “Notably, [21, 39, 46, 49, 56, 57] all use datasets synthesized with the GraspIt! [63] simulator, which is widely used for both multi-finger robotic and human grasp synthesis.”