GenDICE: Generalized Offline Estimation of Stationary Values

By Ruiyi Zhang, Bo Dai, Lihong Li, Dale Schuurmans. 2020

188 citations recorded by the source corpus.

An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove the consistency of the method under general conditions, provide a detailed error analysis, and demonstrate strong empirical performance on benchmark tasks, including off-line PageRank and off-policy policy evaluation.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Only recently have practical methods for estimating the marginalized importance ratio been introduced (Sutton et al., 2016; Yu, 2015; Hallak et al., 2015, 2016; Hallak and Mannor, 2017; Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Recent approaches [36, 17, 40, 60] use marginalized importance sampling by directly estimating the state-distribution importance ratios via some form of dynamic programming [34] and typically exhibit less variance than per-action importance sampling at the cost of bias.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “…2018) is off-policy evaluation (Precup et al., 2001; Maei, 2011; Jiang & Li, 2015; Sutton & Barto, 2018; Liu et al., 2018; Nachum et al., 2019; Zhang et al., 2020), where we want to estimate the performance of a target policy (average reward in the continuing setting or expected total…”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “In fact, existing estimators for OPE in the behavior-agnostic setting which typically reduce the OPE problem to estimation of quantities wπ/D (e.g., DualDICE (Nachum et al., 2019b) and GenDICE (Zhang et al., 2020)) can be recast as special cases by introducing different regularizations to the Lagran”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, methodology): “[12] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans.”
Flexible Option Learning (2021, methodology): “A possible solution would be to estimate this quantity by making use of recent approaches [Hallak and Mannor, 2017, Xie et al., 2019, Zhang et al., 2020].”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, methodology): “Recent work in density ratio estimation for OPPE include a novel family of estimation algorithms called Distribution Correction Estimation (DiCE) (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, unknown): “…Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara and Jiang, 2019, Kallus and Uehara, 2020, Jiang and Huang, 2020, Feng et al., 2020, Yang et al., 2020, Nachum et al., 2019a, Zhang et al., 2020a,b].”
Representation Balancing Offline Model-based Reinforcement Learning (2021, unknown): “Recently, techniques of computing a stationary DIstribution Correction Estimation (DICE) have made remarkable progress that effectively addresses the curse of horizon (Liu et al., 2018a; Nachum et al., 2019a; Tang et al., 2020; Zhang et al., 2020; Mousavi et al., 2020).”
Offline Policy Selection under Uncertainty (2020, methodology): “…value (Dudík et al., 2011; Bottou et al., 2013; Jiang and Li, 2015; Thomas and Brunskill, 2016; Kallus and Uehara, 2020; Nachum et al., 2019; Zhang et al., 2020; Yang et al., 2020), there is also a growing body of literature that uses frequentist principles to derive high-conﬁdence lower…”
Off-Policy Evaluation via the Regularized Lagrangian (2020, methodology): “While behavior-agnostic OPE appears to be a daunting problem, a number of estimators have recently been developed for this scenario [21, 28, 30, 31], demonstrating impressive empirical results.”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “…et al., 2018; Xie et al., 2019; Nachum et al., 2019a;b; Tang et al., 2019; Kallus and Uehara, 2019; 2020; Jiang and Huang, 2020; Uehara et al., 2020; Duan et al., 2020; Yin and Wang, 2020; Yin et al., 2020; Nachum and Dai, 2020; Yang et al., 2020a; Zhang et al., 2020b) and the references therein.”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “This density ratio can be learned by either Liu et al. (2018) as Liu et al. (2019) did or (discounted) COP-TD, Nachum et al. (2019a), Uehara & Jiang (2019), Zhang et al. (2020a), and GradientDICE (Zhang et al., 2020b).”