Accelerating Reinforcement Learning with Learned Skill Priors

By Karl Pertsch, Youngwoon Lee, Joseph J. Lim. 2020

317 citations recorded by the source corpus.

Intelligent agents rely heavily on prior experience when learning a new task, yet most modern reinforcement learning (RL) approaches learn every task from scratch. One approach for leveraging prior knowledge is to transfer skills learned on prior tasks to the new task. However, as the amount of prior experience increases, the number of transferable skills grows too, making it challenging to explore the full set of available skills during downstream learning. Yet, intuitively, not all skills should be explored with equal probability; for example information about the current state can hint which skills are promising to explore. In this work, we propose to implement this intuition by learning a prior over skills. We propose a deep latent variable model that jointly learns an embedding space of skills and the skill prior from offline agent experience. We then extend common maximum-entropy RL approaches to use skill priors to guide downstream learning. We validate our approach, SPiRL (Skill-Prior RL), on complex navigation and robotic manipulation tasks and show that learned skill priors are essential for effective skill transfer from rich datasets. Videos and code are available at this https URL.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches (Ajay et al., 2020; Campos et al., 2020; Pertsch et al., 2020; Singh et al., 2021) or by maximizing mutual information (Eysenba”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “Other approaches use CVAEs [23, 32, 12, 24, 34] to learn a latent embedding to represent the underlying skills.”
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks (2019, background): “, block stacking [28], ball serving [19], and kitchen tasks [29], [30], have been proposed but limited to little variation in shapes and physical properties of objects.”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “…with a large 6-layer transformer, and previous 10 observations as history; Implicit BC : the ofﬁcial implementation of energy-based models for BC (Florence et al., 2022); SPiRL : using a VAE, originally from Pertsch et al. (2021); and PARROT : a ﬂow-based model, originally from Singh et al. (2020).”
Skill-based Meta-Reinforcement Learning (2022, methodology): “Following prior work on learning from large ofﬂine datasets (Lynch et al., 2020; Pertsch et al., 2020; 2021), we assume access to a dataset of state-action trajectories D = { s t , a t , ... } which is collected either across a wide variety of tasks or as “play data” with no particular task in mind.”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[87] Karl Pertsch, Youngwoon Lee, and Joseph J.”
Learning latent actions to control assistive robots (2021, unknown): “Robots can learn low-dimensional models of states [40], dynamics [49,50], movement primitives [39], trajectories [14], plans [32], policies [17], skills [42], and action representations for reinforcement learning [11].”