Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

By Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, Karol Hausman. 2019

559 citations recorded by the source corpus.

We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at this https URL

Subfields: Artificial Intelligence, Computer Science Applications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Reinforcement Learning

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Next, we evaluate CQL on the Franka kitchen domain [18] from D4RL [14].”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “We find that algorithms that use on-policy fine-tuning [37, 12], or Monte-Carlo return estimation [32, 33] are generally much less efficient than off-policy actor-critic algorithms, which iterate between improving π and estimating Q via Bellman backups.”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “[11], involves controlling a 9-DoF Franka robot in a kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “via human teleoperation [28, 9, 20, 19] or any combination of these.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “This technique trains goal conditioned policies to reach any previously visited state upon demand, with many recent examples in RL [29, 14, 28] and IL [26, 8].”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, methodology): “To evaluate the performance of ZeST, we ﬁrst consider an experience dataset in the Franka Kitchen domain (Gupta et al., 2019).”
Skill-based Meta-Reinforcement Learning (2022, methodology): “The FrankaKitchen environment of Gupta et al. (2019) requires the agent to control a 7-DoF robot arm via continuous joint velocity commands and complete a sequence of manipulation tasks like opening the microwave or turning on the stove.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “…policy through goal generation or an “options” framework [6, 8, 59], learning primitives through intrinsic motivation or other auxiliary objectives [60, 61, 62], from demonstration behavior [63, 64, 65, 66, 67], or from low-level, potentially goal conditioned, reward functions [7, 58, 68, 69].”
Interactive Language: Talking to Robots in Real Time (2022, background): “feedback, over the more common “open-loop” evaluation setting where the sequence of subgoals is decided up front [10]–[12], [43].”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “• Relay Imitation Learning (RIL) is a hierarchical policy, that learns a high-level sub-goal generator, which is used to condition a low-level MLP policy [12].”
Behavior Transformers: Cloning k modes with one stone (2022, methodology): “We experimentally evaluate BeT on ﬁve datasets ranging from simple diagnostic toy datasets to complex datasets that include simulated robotic pushing [25], sequential task solving in kitchen environments [34], and self-driving with visual observations in CARLA [21].”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “However, many such approaches are limited to short-horizons [8, 9].”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “Recent works have also considered imitation learning with goal relabeling for learning from human play data [Lynch et al., 2019, Gupta et al., 2019] or demonstrations [Ding et al., 2019].”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “…task-specific features, leading to a significant improvement in the success rate of embodied control tasks, outperforming both R3M [12] (a video-language contrastive learned model) and BLIP-2 [13] (a multi-modal foundation model) on Franka Kitchen [14] and Meta-World [15] environments. data.”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “These were collected using a virtual reality setup, with a human’s movements translated to robot joint actuations (Gupta et al., 2020).”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, methodology): “Evaluation Environments For our simulation experiments, we consider the FrankaKitchen (Gupta et al., 2019) environment, in which a 7-DoF Franka robot is tasked with manipulating common household kitchen objects to pre-speciﬁed conﬁgurations.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “We observe that in the Franka Kitchen and Adroit environments, the performance gain from R3M stays consistent with increase in dataset size, even as the absolute performance of all methods improves.”
You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL (2021, methodology): “We use MuJoCo [32] locomotion tasks [33], Adroit hand-manipulation tasks [34], and Franka kitchen [35] tasks.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “Some recent works utilize the training data or sample from the training data to make the ofﬂine datasets [Wu et al. , 2019; Kumar et al. , 2019; G ¨ ulc¸ehre et al. ].”