DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System

By Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan D. Ratliff, Dieter Fox. 2019

316 citations recorded by the source corpus.

Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually offer reduced degrees of control. Herein, a low-cost, depth-based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enabled operators to solve a variety of complex manipulation tasks that go beyond simple pick-and-place operations and performance was measured through speed and reliability metrics. DexPilot cost-effectively enables the production of high dimensional, multi-modality, state-action data that can be leveraged in the future to learn sensorimotor policies for challenging manipulation tasks. The videos of the experiments can be found at https://sites.google.com/view/dex-pilot.

Subfields: Cognitive Neuroscience, Control and Systems Engineering, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Control and Systems Engineering

In Mechanical Engineering

Citations in the corpus

Visual Imitation Made Easy (2020, background): “However, it is challenging to devise such interfaces for robotic manipulation [24].”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, background): “3D Interaction Models: Learning a model of human-object interactions is useful for graphics and robotics to help avatars [12, 63] or robots [21] interact with their surroundings, and for vision [18, 42] to help reconstruct interactions from ambiguous data.”
Grounding Language in Play (2020, methodology): “This type of data stands in contrast to conventional teleoperated multitask demonstrations (Zhang et al., 2018; Handa et al., 2019; Singh et al., 2020), whose diversity is necessarily constrained by an upfront task deﬁnition.”
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023, background): “Skills such as threading a zip tie, inserting RAM, and juggling ping pong ball, to our knowledge, are not available for existing teleoperation systems with 5-10x the budget [21, 5].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, unknown): “As such, there are substantial ongoing efforts at collecting larger and larger datasets [49], [50].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, methodology): “To bridge the gap between these two models, Hand Motion Retargeting, also called Kinematics Retargeting [25] or Teleportation Mapping [40, 60], can be used to map the observed human hand pose to the robot hand joints.”
Masked Visual Pre-training for Motor Control (2022, background): “A number of works consider related manipulation problems with multi-ﬁnger hands but rely on explicit state estimation (Handa et al., 2020; Huang et al., 2021), expert policies (Chen et al., 2021a), human demonstrations (Ra-jeswaran et al., 2018; Radosavovic et al., 2021; Qin et al., 2021), human…”
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (2024, methodology): “We use VR for the dexterous hand control in this work, but the hand pose estimation could be replaced by other interfaces (e.g., MoCap gloves [54] or RGB cameras [55]) as well.”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, unknown): “The vector optimizers formulate the retargeting problem as an optimization problem [4, 26] while the optimization is defined based on user-selected vectors: In the above formulation, q t denotes the robot joint angles at time t , and v it is the i-th keypoint vector on the human hand.”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “and video [13] or paired video and kinesthetic demos [45], [46].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “While demonstrations may implicitly reveal activity-contexts, they are cumbersome to collect, requiring expert teleoperation, often using specialized hardware (VR, MoCap) in artiﬁcial lab settings [67, 48, 27], and need to be collected independently for each new task.”
DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video (2022, background): “Similarly, advanced visual teleoperation systems that observe the bare hand [13] still separate the person’s hand from the real-world object being manipulated.”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “For example, in robotics, a reliable motion capture for hand manipulation of objects is crucial for both learning from human demonstration [12] and fluent and safe human-robot interaction [46].”