Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

By Xue Bin Peng, Aviral Kumar, Grace Zhang, Sergey Levine. 2019

869 citations recorded by the source corpus.

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Following prior work [29, 28, 17, 15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq.”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “We compare to CQL [17], BEAR [19], BRAC [20], and AWR [25].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “First, most of these methods fit an estimated model of the behavior policy πβ(·|s) from the dataset and constrain the learned policy π(·|s) against this estimated behavior policy (though not all, see for example Peng et al. (2019)).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “If we choose R(µ) to be the KL-divergence against a prior distribution, ρ(a|s), i.e., R(µ) = −DKL(µ, ρ), then we get µ(a|s) ∝ ρ(a|s) ·exp(Q(s,a)) (for a derivation, see Appendix A).”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “Thus, if we assume unlimited capacity and no sampling error, the optimal parameters should satisfy Prior work (Brandfonbrener et al., 2021; Peng et al., 2019) has proposed directly using this objective to learn Q π β , and then train the policy π ψ to maximize Q π β .”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “We find that the design decisions to use off-policy critic estimation allow AWAC to significantly outperform AWR [12] while the implicit behavior modeling allows AWAC to significantly outperform ABM [17], although ABM does make some progress.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “To do so, a wide range of ofﬂine RL methods propose to mitigate bootstrapped error by constraining the learned policy to the behavior policy induced by the dataset [18, 35, 70, 29, 48, 51, 57].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “Finally, we benchmark several state-of-the-art algorithms [13, 18, 37, 2, 7, 28, 29], demonstrating that while these algorithms perform well in the settings they were designed for, they can perform poorly on tasks such as data collected from hand-designed controllers and multi-task behavior.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…with respect to π is where the softmax function is deﬁned as Thus, we see that the use of KL-divergence leads to a dual formulation which bears similarities to max-likelihood policy learning, a common goal of a number of recent works (Abdolmaleki et al., 2018; Song et al., 2019; Peng et al., 2019).”
Offline RL Without Off-Policy Evaluation (2021, background): “…primarily relies on actor-critic style algorithms that alternate between policy evaluation and policy improvement [Fujimoto et al., 2018a, 2019, Peng et al., 2019, Kumar et al., 2019, 2020, Wang et al., 2020b, Wu et al., 2019, Kostrikov et al., 2021, Jaques et al., 2019, Siegel et al., 2020,…”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “, distributional matching [17], support matching [28, 63], imposition of a behavior-based prior [52], or implicit constraints via selective policy-updates [46, 60].”
Critic Regularized Regression (2020, methodology): “By using returns R for advantage estimation (computing advantages as R − V ), however, Advantage weighted regression (AWR) [28], or similar methods, would favor the ﬁrst arm over the second arm; regardless of the temperature parameter.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Prior model-free offline RL algorithms have been designed to regularize the learned policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], offline actor-critic methods [53, 45, 27, 16, 64], applying uncer”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “A number of works have leveraged this insight to develop RL algorithms [3, 29, 31, 32].”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “AWR was proposed mostly as an off-policy RL algorithm.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “Training agents from pre-collected dataset has long been studied (Ho and Ermon, 2016; Kang, Jie, and Feng, 2018; Oh et al., 2018; Peng et al., 2019; Fu et al., 2020; Qin et al., 2021).”