Hand-Object Contact Consistency Reasoning for Human Grasps Generation

By Hanwen Jiang, Shaowei Liu, Jiashun Wang, Xiaolong Wang. 2021

252 citations recorded by the source corpus.

While predicting robot grasps with parallel jaw grippers have been well studied and widely applied in robot manipulation tasks, the study on natural human grasp generation with a multi-finger hand remains a very challenging problem. In this paper, we propose to generate human grasps given a 3D object in the world. Our key observation is that it is crucial to model the consistency between the hand contact points and object contact regions. That is, we encourage the prior hand contact points to be close to the object surface and the object common contact regions to be touched by the hand at the same time. Based on the hand-object contact consistency, we design novel objectives in training the human grasp generation model and also a new self-supervised task which allows the grasp generation network to be adjusted even during test time. Our experiments show significant improvement in human grasp generation over state-of-the-art approaches by a large margin. More interestingly, by optimizing the model during test time with the self-supervised task, it helps achieve larger gain on unseen and out-of-domain objects. Project page: https://hwjiang1510.github.io/GraspTTA/.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Human-Computer Interaction

In Robotics & Control

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “A line of works [2,7,21,23,67] have also attempted to generate hand pose conditioned on objects.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “Notably, [21, 39, 46, 49, 56, 57] all use datasets synthesized with the GraspIt! [63] simulator, which is widely used for both multi-finger robotic and human grasp synthesis.”
Human Hands as Probes for Interactive Object Understanding (2021, background): “This can be in the form of learning a) where and how to grasp [9, 20, 25, 26, 28, 33, 34, 37, 41, 49], b) state classifiers [24], c) interaction hotspots [14, 40, 42, 59], d) spatial priors for action sites [44], e) object articulation modes [11,36], f) reward functions [27,29,48,50], g) functional…”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “[12] propose to collect the contact maps on objects with a thermal camera, which can be used to learn functional grasps [32, 13, 44].”