DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

By Priyanka Mandikal, Kristen Grauman. 2022

139 citations recorded by the source corpus.

Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning. We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interactions present in in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual data. As a result, it can easily scale to new objects, and it sidesteps the standard practice of collecting human demonstrations in a lab -- a much more expensive and indirect way to capture human expertise. Through experiments on 27 objects with a 30-DoF simulated robot hand, we demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment to obtain human demonstrations, while also being faster to train. Project page: https://vision.cs.utexas.edu/projects/dexvip-dexterous-grasp-pose-prior

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Saurabh Gupta

In Robotics & Control

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Other directions of hand-object interaction involve using hands as probes for object understanding [15], affordance hotspot reasoning [47,49], or even leveraging visual hand-object understanding for robotic learning [41,42,48].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “Similarly, hand-grasps afforded by objects and their locations provide priors for exploration [38, 39].”