A Minimalist Approach to Offline Reinforcement Learning

By Scott Fujimoto, Shixiang Shane Gu. 2021

1,199 citations recorded by the source corpus.

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “, 2020], or by adding a supervised learning term to the policy improvement objective [Fujimoto and Gu, 2021].”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Prior model-free offline RL algorithms have been designed to regularize the learned policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], offline actor-critic methods [53, 45, 27, 16, 64], applying uncer”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Adding a BC regularization term (Fujimoto and Gu, 2021) in the actor objective helps mitigate this phenomenon”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “These pre-trained language models acheive state-of-the-art results by outperforming the strong Decision Transformer and TD3+BC baselines by a signiﬁcant 3.0-5.4 points.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Lastly, we compare with two more complex methods that use TD learning: TD3+BC [39] and IQL [68].”
Continuous Control with Action Quantization from Demonstrations (2021, methodology): “Similarly a number of methods in Imitation Learning (Brantley et al., 2019; Wang et al., 2019) or in offline RL (Fujimoto & Gu, 2021; Wu et al., 2019) are evaluated on continuous control tasks and are based on Behavioral Cloning regularization which could be refined using the same type of…”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “To this end, recent methods use implicit divergence constraints [38, 37, 33, 45], explicit density models [47, 13, 27, 15], and supervised learning terms [12].”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, methodology): “As the scale of the Q-value function varies in different offline datasets, to normalize it, we follow Fujimoto & Gu (2021) , where η is a hyperparameter that balances the two loss terms and the Q in the denominator is for normalization only and not differentiated over.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “For (i), we include CQL [27] as well as the more recent TD3+BC [14] and Onestep RL [4] methods.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “RL methods, Fujimoto and Gu [44] propose adding a behaviorcloning regularizer to the policy improvement step of the twin”