Offline RL Without Off-Policy Evaluation

By David Brandfonbrener, William F. Whitney, Rajesh Ranganath, Joan Bruna. 2021

205 citations recorded by the source corpus.

Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “2 (c, d) illustrates the difference between single-step methods, in this case represented by Onep-step RL (Brandfonbrener et al., 2021; Wang et al., 2018) and IQL with τ = 0 .”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “For (i), we include CQL [27] as well as the more recent TD3+BC [14] and Onestep RL [4] methods.”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, result): “This can be interpreted as taking a single step of policy improvement which, though often effective [46], is known to be suboptimal as compared to full dynamic programming methods (i.”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, methodology): “…BC, BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019), BCQ (Fujimoto et al., 2019), TD3+BC (Fujimoto & Gu, 2021), AWR (Peng et al., 2019), AWAC (Nair et al., 2020), and IQL (Kostrikov et al., 2021b), along with the Onestep RL (Brandfonbrener et al., 2021), which is based on single-step improvement.”