Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation

By Dafni Antotsiou, Guillermo Garcia-Hernando, Tae-Kyun Kim. 2018

87 citations recorded by the source corpus.

Human hand actions are quite complex, especially when they involve object manipulation, mainly due to the high dimensionality of the hand and the vast action space that entails. Imitating those actions with dexterous hand models involves different important and challenging steps: acquiring human hand information, retargeting it to a hand model, and learning a policy from acquired data. In this work, we capture the hand information by using a state-of-the-art hand pose estimator. We tackle the retargeting problem from the hand pose to a 29 DoF hand model by combining inverse kinematics and PSO with a task objective optimisation. This objective encourages the virtual hand to accomplish the manipulation task, relieving the effect of the estimator’s noise and the domain gap. Our approach leads to a better success rate in the grasping task compared to our inverse kinematics baseline, allowing us to record successful human demonstrations. Furthermore, we used these demonstrations to learn a policy network using generative adversarial imitation learning (GAIL) that is able to autonomously grasp an object in the virtual space.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “Some researchers have successfully utilized expert human demonstrations in guiding the training process of the agents [17, 63, 4, 122].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, methodology): “Our hand joint retargeting approach is inspired by [3, 25].”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “Grasp synthesis is a longstanding problem in robotics and graphics, resulting in an extensive literature [1, 4, 6, 16, 36, 40, 43, 47, 59, 65, 70, 76, 77, 81, 95].”
DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System (2019, methodology): “[20] show teleoperation of an anthropomorphic hand in simulation using depth based hand tracking.”