Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

By Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, Sergey Levine. 2020

172 citations recorded by the source corpus.

Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches (Ajay et al., 2020; Campos et al., 2020; Pertsch et al., 2020; Singh et al., 2021) or by maximizing mutual information (Eysenba”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “However, such concept is still relatively fresh in RL (Singh et al., 2020; Tirumala et al., 2020), due to the difﬁculty in parameterizing different scenes and tasks through a single network (Wang et al., 2018b; Jiang et al., 2019; Zeng et al., 2020) as well as the lack of large off-the-shelf…”
Continuous Control with Action Quantization from Demonstrations (2021, methodology): “Singh et al. (2020) also exploit the demonstrations prior for downstream tasks by learning a prior using a state-conditioned action generative model coupled with a continuous action algorithm.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “In addition, the asymptotic results that safe RL methods can achieve are summarised in the preprint version of the paper [76].”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, background): “Normalizing flows have also been proposed as a policy representation [36].”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “…with a large 6-layer transformer, and previous 10 observations as history; Implicit BC : the ofﬁcial implementation of energy-based models for BC (Florence et al., 2022); SPiRL : using a VAE, originally from Pertsch et al. (2021); and PARROT : a ﬂow-based model, originally from Singh et al. (2020).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “They collected safe trajectories as a safe set by a scripted policy [255], and applied the safe trajectories to a learning process.”