Offline Reinforcement Learning with Fisher Divergence Critic Regularization

By Ilya Kostrikov, Jonathan Tompson, Rob Fergus, Ofir Nachum. 2021

359 citations recorded by the source corpus.

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data. In this work, we propose an alternative approach to encouraging the learned policy to stay close to the data, namely parameterizing the critic as the log-behavior-policy, which generated the offline data, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. We propose using a gradient penalty regularizer for the offset term and demonstrate its equivalence to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature. We thus term our resulting algorithm Fisher-BRC (Behavior Regularized Critic). On standard offline RL benchmarks, Fisher-BRC achieves both improved performance and faster convergence over existing state-of-the-art methods.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning with Implicit Q-Learning (2021, background): “Several works have also proposed to directly regularize the Q-function to produce low values for out-of-distribution actions (Kostrikov et al., 2021; Kumar et al., 2020).”
Implicit Behavioral Cloning (2021, methodology): “, value functions) for various reinforcement learning (RL) methods [13, 14, 15, 16], our work presents a distinct yet simple method: do BC with implicit models.”
Offline RL Without Off-Policy Evaluation (2021, background): “…the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state, action pairs [Wu et al., 2019, Nachum et al., 2019, Kumar et al., 2020, Kostrikov et al., 2021, Gulcehre et al., 2021].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Our re-implementation of FisherBRC (in PyTorch rather than Tensorflow) is used only for run time experiments.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], ofﬂine actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantiﬁcation to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29, 55] or…”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Prior works (Kumar et al., 2020; Kostrikov et al., 2022; Wu et al., 2019; Kostrikov et al., 2021; Dadashi et al., 2021; Ajay et al., 2020; Ghosh et al., 2022) have tackled this problem with reinforcement learning (RL) that uses dynamic programming for trajectory stitching.”
Behavior Transformers: Cloning k modes with one stone (2022, background): “Finally, a class of algorithms, most notably [50, 25, 43, 58] do not directly learn a generative model but instead learn energy based models.”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “The state value function is trained via expectile regression onto the Q-values implicitly performing a maximization over actions.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “[38] proposed Fisher-behavior regularized critic (Fisher-BRC), which uses a Fisher diver-”