Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

By Ajay Mandlekar, Danfei Xu, Roberto Mart'in-Mart'in, Silvio Savarese, Li Fei-Fei. 2020

177 citations recorded by the source corpus.

Imitation learning is an effective and safe technique to train robot policies in the real world because it does not depend on an expensive random exploration process. However, due to the lack of exploration, learning policies that generalize beyond the demonstrated behaviors is still an open challenge. We present a novel imitation learning framework to enable robots to 1) learn complex real world manipulation tasks efficiently from a small number of human demonstrations, and 2) synthesize new behaviors not contained in the collected demonstrations. Our key insight is that multi-task domains often present a latent structure, where demonstrated trajectories for different tasks intersect at common regions of the state space. We present Generalization Through Imitation (GTI), a two-stage offline imitation learning algorithm that exploits this intersecting structure to train goal-directed policies that generalize to unseen start and goal state combinations. In the first stage of GTI, we train a stochastic policy that leverages trajectory intersections to have the capacity to compose behaviors from different demonstration trajectories together. In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations. We validate GTI in both simulated domains and a challenging long-horizon robotic manipulation domain in the real world. Additional results and videos are available at this https URL .

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “We chose to evaluate 6 algorithms in this study – Behavioral Cloning (BC), BC with an RNN policy (BC-RNN), Hierarchical Behavioral Cloning (HBC) [10], Batch-Constrained Q-Learning (BCQ) [26], Conservative Q-Learning (CQL) [27], and IRIS [20].”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, background): “Recent works [7, 19, 26, 44] use deep neural networks to map directly from image observations to action, and have proven effective for learning visualmotor skills for complex and diverse manipulation tasks.”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “Hierarchical Behavioral Cloning (HBC) seeks to learn hierarchical policies that encourage temporal abstraction and can be a better way to learn from offline human demonstrations [25, 26].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, methodology): “A policy can be learned without any interaction, as in Behavioral Cloning (BC) [27, 33, 41] or using additional interaction with the environment, as in Inverse Reinforcement Learning (IRL) [1, 2].”
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024, methodology): “It has been used extensively in prior works [49, 31, 9, 2, 16, 6, 17].”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “While conceptually simple, behavior cloning has shown surprising promise on an array of real-world robot tasks, including manipulation [60, 13, 28, 27, 59, 37, 4] and autonomous driving [36, 6].”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “Other approaches use CVAEs [23, 32, 12, 24, 34] to learn a latent embedding to represent the underlying skills.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, methodology): “[25] Ajay Mandlekar, Danfei Xu, Roberto Martı́n-Martı́n, Silvio Savarese, and Li Fei-Fei.”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, background): “Our idea of learning a hierarchical policy from offline demonstrations is also related to prior works [29, 31, 32, 49, 59].”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, methodology): “Pre-trained ResNet on image classification task on ImageNet (Deng et al., 2009) has been a popular and very effective embedding model for extracting visual features for down-stream robotic manipulation tasks (Gupta et al., 2018; Mandlekar et al., 2020; Shah and Kumar, 2021).”
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, background): “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3–7,14,18,19,22,30,33,51, 55, 56, 64–66], making the system broadly applicable.”