GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes

By Enric Corona, Albert Pumarola, Guillem Alenyà, Francesc Moreno-Noguer, Grégory Rogez. 2020

216 citations recorded by the source corpus.

The rise of deep learning has brought remarkable progress in estimating hand geometry from images where the hands are part of the scene. This paper focuses on a new problem not explored so far, consisting in predicting how a human would grasp one or several objects, given a single RGB image of these objects. This is a problem with enormous potential in e.g. augmented reality, robotics or prosthetic design. In order to predict feasible grasps, we need to understand the semantic content of the image, its geometric structure and all potential interactions with a hand physical model. To this end, we introduce a generative model that jointly reasons in all these levels and 1) regresses the 3D shape and pose of the objects in the scene; 2) estimates the grasp types; and 3) refines the 51-DoF of a 3D hand model that minimize a graspability loss. To train this model we build the YCB-Affordance dataset, that contains more than 133k images of 21 objects in the YCB-Video dataset. We have annotated these images with more than 28M plausible 3D human grasps according to a 33-class taxonomy. A thorough evaluation in synthetic and real images shows that our model can robustly predict realistic grasps, even in cluttered scenes with multiple objects in close contact.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “A line of works [2,7,21,23,67] have also attempted to generate hand pose conditioned on objects.”
Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, methodology): “Generating human grasp is very challenging due to the higher degree-of-freedom of the human hand [24, 9, 46, 3, 21].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “One problem is to synthesize realistic grasp poses for generic objects [4].”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “GANHand [15] learns to predict human grasp affordances (as poses of a MANO [75] hand model) from input RGBD images using GANs.”
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, methodology): “Instead of hand-crafted contact constraints [8, 19, 58] during optimization, we use data-driven constraints generated from GNet.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “Recently, there has been increased interest in hand-object reconstruction from RGB images [12, 21, 24, 25, 40, 66, 77, 86].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Datasets of hand grasping scenario usually require manual annotations [43, 6] or depth tracking [51] to obtain the ground truth.”
Human Hands as Probes for Interactive Object Understanding (2021, methodology): “We adapt the YCB-Affordance benchmark [9] for afforded hand-grasp prediction.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Dynamic human object interactions Recently, there has been a strong push on modeling hand-object interactions based on 3D [42,72], 2.5D [13,14] and 2D [22,24,28,35,83] Dataset RGBD Hum.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Another line of datasets focus on hand-object contact reasoning [12, 17, 78, 14].”