AlgaeDICE: Policy Gradient from Arbitrary Experience

By Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, Dale Schuurmans. 2019

264 citations recorded by the source corpus.

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…of the f-divergence shown below Df (p, q) = max x:S×A→R ( Ey∼p(y)[x(y)]− Ey∼q(y)[f∗(x(y))] ) , (14) and then applying the change of variables trick from Nachum et al. (2019a) discussed previously, we can obtain a saddle-point optimization problem for optimizing the regularized RL objective, max…”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “Finally, we benchmark several state-of-the-art algorithms [13, 18, 37, 2, 7, 28, 29], demonstrating that while these algorithms perform well in the settings they were designed for, they can perform poorly on tasks such as data collected from hand-designed controllers and multi-task behavior.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “14See the parallel work by Nachum et al. (2019) and the followup work of Jiang and Huang (2020) for some intriguing discussions on this matter.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Other works, such as AlgaeDICE (Nachum et al., 2019) and CQL (Kumar et al., 2020) attempt to incorporate some divergence regularization into the critic.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], ofﬂine actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantiﬁcation to the predictions of the Q-values [2, 28, 63, 34], and…”
Experience Replay with Likelihood-free Importance Weights (2020, background): “Dual representations of f -divergences are also leveraged in reinforcement learning (Nachum et al., 2019; Nachum and Dai, 2020), but it is used over a regularizer that encourages exploration to be closer to off-policy experiences; the importance weights are added to the reward function when computin”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara and Jiang, 2019, Kallus and Uehara, 2020, Jiang and Huang, 2020, Feng et al., 2020, Yang et al., 2020, Nachum et al., 2019a, Zhang et al., 2020a,b].”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “See, e.g., (Jiang and Li, 2016; Thomas and Brunskill, 2016; Farajtabar et al., 2018; Liu et al., 2018; Xie et al., 2019; Nachum et al., 2019a;b; Tang et al., 2019; Kallus and Uehara, 2019; 2020; Jiang and Huang, 2020; Uehara et al., 2020; Duan et al., 2020; Yin and Wang, 2020; Yin et al., 2020;…”
Offline Reinforcement Learning from Images with Latent Space Models (2020, methodology): “These include regularized variants of importance sampling based algorithms (Liu et al., 2019; Swaminathan and Joachims, 2015; Nachum et al., 2019; Zhang et al.)”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “Nachum et al. (2019b) show the primal variable works similarly to an actor, and the dual variable works similarly to a critic.”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, methodology): “To this end, we consider the following KL-regularized ofﬂine RL objective (Nachum et al., 2019) for some to-be-speciﬁed reward r ( o, g ) : where d π H ( o, a H ; g ) is the distribution over observations and actions π H visits conditioned on g .”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “We used the implementation provided by the authors in case of DualDICE.”