OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

By Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum. 2020

195 citations recorded by the source corpus.

Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning. Primitives extracted in this way serve two purposes: they delineate the behaviors that are supported by the data from those that are not, making them useful for avoiding distributional shift in offline RL; and they provide a degree of temporal abstraction, which reduces the effective horizon yielding better learning in theory, and improved offline RL in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning as well as exploration and transfer in online RL on a variety of benchmark domains. Visualizations are available at this https URL

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches (Ajay et al., 2020; Campos et al., 2020; Pertsch et al., 2020; Singh et al., 2021) or by maximizing mutual information (Eysenba”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “While some works have proposed learning action representations [8, 30], our work focuses on state representation learning, which is more common in the literature, and whose aim is generally to distill aspects of the observation relevant to control from those relevant only to measurement [3]; see [27”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, neutral): “This broad and general question has been partially answered by previous works (Ajay et al., 2020; Singh et al., 2020).”
Skill-based Meta-Reinforcement Learning (2022, background): “When trained from diverse datasets these approaches can extract a wide repertoire of skills and learn complex, long-horizon tasks (Merel et al., 2020; Lynch et al., 2020; Pertsch et al., 2020; Ajay et al., 2021; Chebotar et al., 2021; Pertsch et al., 2021).”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “To solve these issues, offline RL is an excellent choice to deal with practical problems [3, 24, 35, 42, 15, 28, 4, 23, 54, 12], aiming at learning from a fixed dataset without interaction with environments.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, background): “Some algorithms have also proposed to learn a state conditioned VAEs to minimize the extrapolation error (Fujimoto et al., 2019; Kumar et al., 2019; Ajay et al., 2020).”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “Prior works (Kumar et al., 2020; Kostrikov et al., 2022; Wu et al., 2019; Kostrikov et al., 2021; Dadashi et al., 2021; Ajay et al., 2020; Ghosh et al., 2022) have tackled this problem with reinforcement learning (RL) that uses dynamic programming for trajectory stitching.”