Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

By Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Martin A. Riedmiller. 2020

303 citations recorded by the source corpus.

Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “To mitigate the impact of distribution shift in offline RL, prior algorithms either (a) constrain the policy action space [29, 30, 31] or (b) incorporate value pessimism [29, 14], or (c) incorporate pessimism into learned dynamics models [32, 33].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “The KL-divergence constraint can also be enforced implicitly, as in the case of AWR (Peng et al., 2019) and ABM (Siegel et al., 2020).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Prior works have attempted to solve this problem by constraining the learned policy to be “close” to the behavior policy, for example as measured by KL-divergence [27, 59, 48, 54], Wasserstein distance [59], or MMD [30], and then only using actions sampled from this constrained policy in the Bellman”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “Many model-free batch RL methods are designed with two main ingredients: (1) constraining the learned policy to be closer to the behavioral policy either explicitly [19, 36, 72, 30, 49] or implicitly [52, 58], and (2) applying uncertainty quantification techniques, such as ensembles, to stabilize Q-”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Most algorithms from the literature fall into this category [Fujimoto et al., 2018a, Kumar et al., 2019, 2020, Wu et al., 2019, Wang et al., 2020b, Siegel et al., 2020].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…estimate of the behavior policy and then defining an explicit policy parameterization [Fujimoto et al., 2019b, Ghasemipour et al., 2020] or by using divergence regularization [Jaques et al., 2019, Kumar et al., 2019, Wu et al., 2019, Siegel et al., 2020, Guo et al., 2021, Kostrikov et al., 2021].”
Regularized Behavior Value Estimation (2021, background): “…learned policy to stay close to the behavior policy either explicitly or implicitly (Fujimoto et al., 2019b; Kumar et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Wang et al., 2020; Ghasemipour et al., 2020), 2) value-based approaches lower value estimates for unseen state-actions pairs,…”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “…to remain in the vicinity of πb via, e.g., distributional matching (Fujimoto et al., 2019), support matching (Kumar et al., 2019; Wu et al., 2019), imposition of a behaviorbased prior (Siegel et al., 2020), or implicit constraints via selective policy-updates (Wang et al., 2020; Peng et al., 2019).”
Critic Regularized Regression (2020, methodology): “As discussed in recent work, [e.g. 10, 17, 31], without environment interaction (the ofﬂine RL setting) this loss may be problematic due to the “bootstrapping” issue mentioned above; where we evaluate the value of the next state s t +1 by E a ∼ π ( s t +1 ) Q θ (cid:48) ( s t +1 , a ) .”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Datasets and policies We train four offline RL algorithms (D4PG (Barth-Maron et al., 2018), ABM (Siegel et al., 2020), CRR (Wang et al., 2020) and behavior cloning), varying their hyperparameters.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], ofﬂine actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantiﬁcation to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29,…”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “Recently, action priors have been used to leverage offline experience for learning downstream tasks [31].”
Hyperparameter Selection for Offline Reinforcement Learning (2020, background): “Recently, Agarwal et al. (2019), Fujimoto et al. (2019), Kumar et al. (2019) and Siegel et al. (2020) have proposed new oﬄine-RL algorithms and shown that they outperform oﬀ-the-shelf oﬀ-policy RL methods.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “This is in contrast to many previous ofﬂine RL algorithms that augment the value function with a penalty of explicit KL divergence (Siegel et al., 2020; Wu et al., 2019) or maximum mean discrepancy (Kumar et al., 2019).”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “AWR [30] and ABM [36] attempt to estimate the value function of the behavior policy via Monte-Carlo or TD(λ).”