Behavior Regularized Offline Reinforcement Learning

By Yifan Wu, George Tucker, Ofir Nachum. 2019

863 citations recorded by the source corpus.

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “, 2019b), BRAC (Wu et al., 2019), and AWR (Peng et al.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “We compare against ﬁve other methods spanning other approaches to data-driven control: (1) behavior-regularized actor-critic (BRAC; Wu et al. 2019) 2021)) is a concurrently-developed sequence-modeling approach that uses return-conditioning instead of planning; and behavior-cloning (BC) provides the…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “BEAR (Kumar et al., 2019a) uses the MMD distance to represent the policy constraint, while Wu et al. (2019a) evaluates an instantiation of the first order Wasserstein distance.”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “We compare to: prior offline RL methods that use a policy constraint – BEAR [30] and BRAC [59]; SAC [19], an off-policy actor-critic method that we adapt to offline setting; and behavioral cloning (BC).”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “Prior methods generally address this by either constraining the policy to limit how far it deviates from the behavior policy (Fujimoto et al., 2019; Wu et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019; Nair et al., 2020; Wang et al., 2020), or by regularizing the learned value functions to…”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “Choosing the reverse KL results in explicit penalty methods [20] that rely on evaluating the density of a learned behavior model.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “Lastly, uncertainty estimation techniques, such as bootstrap ensembles, are well developed for supervised learning methods [39, 34, 59] and are known to perform poorly for value-based RL methods [70].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “This is similar to evaluation protocols followed in prior work [18, 15, 16].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “While recent work has investigated a number of possible technical reasons for this [Wu et al., 2019], one of the major challenges in addressing these issues has been the lack of realistic evaluation protocols.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Our work bears similarity to the recent body of literature on batch, or offline reinforcement learning [30, 49, 50].”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Another common idea proposed in the literature is to regularize towards the behavior policy [Wu et al., 2019, Jaques et al., 2019, Kumar et al., 2019].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Our offline RL baselines include two state-of-the-art algorithms, CQL [Kumar et al., 2020] and FisherBRC [Kostrikov et al., 2021], as well as BRAC [Wu et al., 2019] and AWAC [Nair et al., 2020] due to their algorithmic simplicity.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Our work adds to the rich literature on behavior regularization methods in offline RL (Wu et al., 2019), which propose a number of regularizations in RL training that compel the learned policy to stay close to the offline data.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “In particular, our method, like [60, 63], only needs a dataset consisting of a single-step transitions and does not require complete episode trajectories.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “Similar to prior works (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019), we train K Q functions (represented by MLPs) and make use of an ensembling procedure to combat overestimation bias (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018b).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “In the paper, we present conservative ofﬂine model-based policy optimization (COMBO), a model-based ofﬂine RL algorithm that penalizes the Q-values evaluated on out-of-support state-action pairs.”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “Notably, although the original BRAC paper uses diﬀerent regularization strengths and policy learning rates for diﬀerent domains, we ﬁx these to values which we found to generally perform best (regularization strength of 1 .”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “Recent works in offline RL (Kumar et al., 2019; Levine et al., 2020; Wu et al., 2019; Ghasemipour et al., 2020; Jaques et al., 2019; Fujimoto et al., 2018) constrain the policy to be close to the data distribution to avoid the use of out-of-distribution actions (Kumar et al.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “Learned behavior priors are commonly used to guide task learning in offline RL approaches [7, 13, 38] in order to avoid value overestimation for actions outside of the training data distribution.”