DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections

By Ofir Nachum, Yinlam Chow, Bo Dai, Lihong Li. 2019

373 citations recorded by the source corpus.

In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correction terms which quantify the likelihood that the new policy will experience a certain state-action pair normalized by the probability with which the state-action pair appears in the dataset -- can improve accuracy and performance. In this work, we propose an algorithm, DualDICE, for estimating these quantities. In contrast to previous approaches, our algorithm is agnostic to knowledge of the behavior policy (or policies) used to generate the dataset. Furthermore, it eschews any direct use of importance weights, thus avoiding potential optimization instabilities endemic of previous methods. In addition to providing theoretical guarantees, we present an empirical study of our algorithm applied to off-policy policy evaluation and find that our algorithm significantly improves accuracy compared to existing techniques.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…of the f-divergence shown below Df (p, q) = max x:S×A→R ( Ey∼p(y)[x(y)]− Ey∼q(y)[f∗(x(y))] ) , (14) and then applying the change of variables trick from Nachum et al. (2019a) discussed previously, we can obtain a saddle-point optimization problem for optimizing the regularized RL objective, max…”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Recent approaches [36, 17, 40, 60] use marginalized importance sampling by directly estimating the state-distribution importance ratios via some form of dynamic programming [34] and typically exhibit less variance than per-action importance sampling at the cost of bias.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “due to error accumulation with distribution shift [8, 21]. Ofﬂine RL methods aim to address this issue, often by constraining the actor updates to avoid excessive deviation from the data distribution [23, 41, 13, 14, 12, 21, 8, 22, 38, 26, 47]. One class of these methods utilize importance sampling ”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[44] perform planning with learned importance weights [45, 46, 47] while using a notion of pessimism for regularization.”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “We compare our approach with [21] and [30] that are two state-of-the-art methods in the oﬀ-policy estimation problem.”
Batch Stationary Distribution Estimation (2020, methodology): “Assumption 2 follows previous work (Liu and Lee, 2017; Nachum et al., 2019), and is common in density ratio estimation (Sugiyama et al., 2008; Gretton et al., 2009) and off-policy evaluation (Wang et al., 2017; Xie et al., 2019).”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “To bound the est, we follow similar arguments by Dai et al. (2018); Nachum et al. (2019) via the covering number.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “Reacher-v2 is from Open AI gym (Brockman et al., 2016) 2 , which is used as a benchmark in both Nachum et al. (2019) and Zhang et al. (2020).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “Applying this change of variables to (10) (after some telescoping, see Nachum et al. (2019b)) yields max The resulting objective is now completely oﬀ-policy, relying only on access to samples from the initial state distribution µ 0 and the oﬀ-policy dataset d D .”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “, 2018, with estimated behavior policy) and DualDICE (Nachum et al., 2019a).”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “We also these to DualDICE (Nachum et al. 2019), which is a variant of the MIS estimator.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “through a KL divergence penalty [20, 47] or clipping actions [9]), or estimating the learned policy’s stationary distribution and using this estimate when performing an RL update [28, 11, 35].”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “We use the implementation from Kostrikov & Nachum (2020), which uses self-normalized (also known as weighted) step-wise importance sampling (Liu et al., 2018; Nachum et al., 2019).”
Hyperparameter Selection for Offline Reinforcement Learning (2020, background): “More recently, marginal importance sampling methods have been developed that directly estimate the stationary state distribution ratio under the two policies (Liu et al., 2018; Nachum et al., 2019).”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “Starting with IH, this style of minimax estimator has recently attracted significant attention in OPE literature, including state-action extension of IH [54, 25] and DICE family of estimators [40, 61, 59, 10].”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “[222] propose the DCRL method that is to optimise the reward and cost from the perspective of an optimisation criterion based on optimisation theory, in which they consider the safety constraints via the duality property [65, 196, 197, 268] with regard to the state density functions, rather than the”