Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

By Aviral Kumar, Justin Fu, George Tucker, Sergey Levine. 2019

1,322 citations recorded by the source corpus.

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “Such errors can then accumulate through the Bellman backup operator [29].”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Because the Bellman equation bootstraps each Q-value based on all future Q estimates, any error can accumulate to distort Q-values (Kumar et al., 2019).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…that serve to constrain the learned policy to avoid out-of-2 distribution behavior that is not supported under the dataset (Fujimoto et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021); imitation learning, where it is used to ﬁt an expert’s actions to obtain a policy (Ross & Bagnell,…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “s have observed several variants of this problem. Fujimoto et al. (2018) noted that restricting Q-value evaluation only to actions in the dataset avoids erroneous Q-values due to extrapolation error. Kumar et al. (2019a) described the out-of-distribution action problem in terms of distributional shi”
Offline Reinforcement Learning with Implicit Q-Learning (2021, background): “The constraints can be implemented via an explicit density model (Wu et al., 2019; Fujimoto et al., 2019; Kumar et al., 2019), implicit divergence constraints (Nair et al., 2020; Wang et al., 2020; Peters & Schaal, 2007; Peng et al., 2019), or by adding a supervised learning term to the policy…”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “While prior methods for fully offline RL provide a mechanism for utilizing offline data [5, 9], as we will show in our experiments, such methods generally are not effective for fine-tuning with online data as they are often too conservative.”
CoMPS: Continual Meta Policy Search (2021, background): “PEARL generally performs poorly across all tasks: although SAC is an off-policy algorithm, it is well known that such methods do not perform well when they are not allowed to gather any additional online data (as, for example, in the case of offline RL) (Kumar et al., 2019).”
Off-policy Bandits with Deficient Support (2020, methodology): “For value function based approaches [10, 14], the objective is based on some estimate of the value function, either through fitting an MDP model and evaluating the value function based on the estimated MDP [10] (a.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “…RL methods are designed with two main ingredients: (1) constraining the learned policy to be closer to the behavioral policy either explicitly [18, 35, 70, 29, 48] or implicitly [51, 57], and (2) applying uncertainty quantiﬁcation techniques, such as ensembles, to stabilize Q-functions [1, 35,…”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “This is similar to evaluation protocols followed in prior work [18, 15, 16].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “…those from deterministic expert policies, are particularly problematic for offline RL algorithms and may cause divergence both empirically [Fujimoto et al., 2018, Kumar et al., 2019] and theoretically [Munos, 2003, Farahmand et al., 2010, Kumar et al., 2019, Agarwal et al., 2019a, Du et al., 2020].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “While [30] show that this choice can be motivated from the perspective of error propagation, we note that there are clear differences between our work and such prior works in batch RL.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, methodology): “In contrast the other methods learn stably in the high-data regime, with BCQ lagging behind in Hopper and Quadruped (sticking too close to the VAE prior actions, an effect already observed in (Kumar et al., 2019)).”
A Framework for Data-Driven Robotics (2019, background): “This is an active area of research, with a number of recent works aimed at improving its stability [63]–[66].”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “We brieﬂy review two recently proposed approaches, BEAR (Kumar et al., 2019) and BCQ (Fujimoto et al., 2018a).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “[32] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine.”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “Our future work in priority addresses this issue, but also the multi-batch setting, when there are several sequential updates [12], extending the method to continuous action spaces [9], and investigating the use of SPIBB in a full online setting, as a value estimation stabilizer.”