Visual Imitation Made Easy

By Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Kumar Gupta, Pieter Abbeel, Lerrel Pinto. 2020

163 citations recorded by the source corpus.

Visual imitation learning provides a framework for learning complex manipulation behaviors by leveraging human demonstrations. However, current interfaces for imitation such as kinesthetic teaching or teleoperation prohibitively restrict our ability to efficiently collect large-scale data in the wild. Obtaining such diverse demonstration data is paramount for the generalization of learned skills to novel scenarios. In this work, we present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. To extract action information from these visual demonstrations, we use off-the-shelf Structure from Motion (SfM) techniques in addition to training a finger detection network. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task. For both tasks, we use standard behavior cloning to learn executable policies from the previously collected offline demonstrations. To improve learning performance, we employ a variety of data augmentations and provide an extensive analysis of its effects. Finally, we demonstrate the utility of our interface by evaluating on real robotic scenarios with previously unseen objects and achieve a 87% success rate on pushing and a 62% success rate on stacking. Robot videos are available at this https URL.

Subfields: Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Robotics & Control

Citations in the corpus

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Prior works on imitation learning have shown one-shot or zero-shot generalization to new objects [1, 2, 3, 4, 5] and to new object goal configurations [6, 7].”
A Framework for Efficient Robotic Manipulation (2020, methodology): “BC has been successfully applied across diverse modalities including video games [35], autonomous navigation [36], [37], autonomous aviation [38], locomotion [39], [40], and manipulation [26], [28], [30], [41].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Unlike previous imitation learning studies with 2-DoF grippers [86, 75], we need the human video to guide the 30-DoF robot hand to move each finger in 3D space.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Many of these works focus on collecting and learning from robot data itself [55, 56, 7, 8, 9, 10, 57].”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, background): “…Yan et al., 2020, Finn et al., 2015, Pinto et al., 2016, Agrawal et al., 2016] and other auxiliary objectives [Jaderberg et al., 2016, Zhan et al., 2020, Young et al., 2020, Chen et al., 2020] have shown improvements on a variety of problems ranging from gameplay, continuous control, and robotics.”
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets (2021, background): “For example, Young et al. [25] presents results on data across many more domains than our bridge data, collected via a hand-held gripper, but only presents two grasping tasks.”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, background): “As a result, despite achieving impressive visual diversity across hundreds of environments, the collected actions are constrained to simple grasping [41] or quasi-static pick-and-place [50, 36], lacking action diversity .”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “By doing so, prior methods have demonstrated how robots can generalize to novel object instances (Finn and Levine, 2017; Levine et al., 2018; Mahler et al., 2017; Pinto and Gupta, 2016; Young et al., 2021), to tasks involving novel combinations of objects and skills (Dasari and Gupta, 2021; Finn et ”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “Finally, there are some works that leverage cheap, off-the-shelf tools, such as reacher-grabber tools, for data collection, equipping robots with the same tools to allow for zero-shot transfer to the robot [47, 53, 63].”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, background): “Other projects have tried using low-cost, handheld grasping devices to collect large datasets [34, 35], but this technique requires inferring actions from video and then transferring behaviors to a robot.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, methodology): “First and third person human demonstrations have been used to train policies directly, transferred either via a handheld gripper [82,108,121] or using online adaptation [6].”