GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values

By Shangtong Zhang, Bo Liu, Shimon Whiteson. 2020

110 citations recorded by the source corpus.

We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020), the state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so any primal-dual algorithm is not guaranteed to converge or find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE's original formulation of the optimization problem. In GradientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE's use of divergence. Consequently, nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Only recently have practical methods for estimating the marginalized importance ratio been introduced (Sutton et al., 2016; Yu, 2015; Hallak et al., 2015, 2016; Hallak and Mannor, 2017; Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “Starting with IH, this style of minimax estimator has recently attracted significant attention in OPE literature, including state-action extension of IH [54, 25] and DICE family of estimators [40, 61, 59, 10].”
Off-Policy Evaluation via the Regularized Lagrangian (2020, methodology): “While behavior-agnostic OPE appears to be a daunting problem, a number of estimators have recently been developed for this scenario [21, 28, 30, 31], demonstrating impressive empirical results.”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “This density ratio can be learned by either Liu et al. (2018) as Liu et al. (2019) did or (discounted) COP-TD, Nachum et al. (2019a), Uehara & Jiang (2019), Zhang et al. (2020a), and GradientDICE (Zhang et al., 2020b).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, unknown): “…Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara and Jiang, 2019, Kallus and Uehara, 2020, Jiang and Huang, 2020, Feng et al., 2020, Yang et al., 2020, Nachum et al., 2019a, Zhang et al., 2020a,b].”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, background): “RL [25], [26], [53], which we will not cover in this survey for brevity and since Levine et al.”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, methodology): “[13] Shangtong Zhang, Bo Liu, and Shimon Whiteson.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “Finally, we define the discounted state occupancy measure as [55, 142] where p πt ( s ) is the probability density over states that policy π visits after t steps.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “The first category involves policy regularizers or constraints to ensure closeness between the learned policy and the behavior policy [20], [21], [47]–[49], [71]–[78].”