Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift

By Carles Gelada, Marc G. Bellemare. 2019

102 citations recorded by the source corpus.

In this paper we revisit the method of off-policy corrections for reinforcement learning (COP-TD) pioneered by Hallak et al. (2017). Under this method, online updates to the value function are reweighted to avoid divergence issues typical of off-policy learning. While Hallak et al.’s solution is appealing, it cannot easily be transferred to nonlinear function approximation. First, it requires a projection step onto the probability simplex; second, even though the operator describing the expected behavior of the off-policy learning algorithm is convergent, it is not known to be a contraction mapping, and hence, may be more unstable in practice. We address these two issues by introducing a discount factor into COP-TD. We analyze the behavior of discounted COP-TD and find it better behaved from a theoretical perspective. We also propose an alternative soft normalization penalty that can be minimized online and obviates the need for an explicit projection step. We complement our analysis with an empirical evaluation of the two techniques in an off-policy setting on the game Pong from the Atari domain where we find discounted COP-TD to be better behaved in practice than the soft normalization penalty. Finally, we perform a more extensive evaluation of discounted COP-TD in 5 games of the Atari domain, where we find performance gains for our approach.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Covariance-shift-based methods have been adapted to the off-policy deep RL setting to deal with the issue of value divergence [19].”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Others have tackled this problem using deep learning, but have not used KL-control (Liu et al., 2019; Gelada and Bellemare, 2019; Bhatt et al., 2019; Kumar et al., 2019; Agarwal et al., 2019; Fujimoto et al., 2018; Ghasemipour et al., 2020).”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Although only non-terminating MDPs have stationary distributions, a state distribution for terminating MDPs with similar properties exists (Gelada & Bellemare, 2019).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Gelada and Bellemare (2019) also discusses several practical tricks, such as soft-normalization and discounted evaluation, to adapt these methods to deep Q-learning settings, unlike prior work that mainly focuses on linear function approximation.”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “A few authors explored the alternative to estimate the propensity score of a state’s stationary distribution [8, 21], when behavior policies are known.”
Batch Stationary Distribution Estimation (2020, background): “It is sufﬁcient to adjust observed rewards according to the ratio between the target and behavior stationary distributions (Hallak and Mannor, 2017; Liu et al., 2018; Gelada and Bellemare, 2019).”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “By rewriting the accumulated reward as an expectation w.r.t. a stationary distribution, Liu et al. (2018); Gelada & Bellemare (2019) recast OPE as estimating a correction ratio function, which significantly alleviates variance.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “Inspired by Hallak & Mannor (2017); Liu et al. (2018), Gelada & Bellemare (2019); Uehara & Jiang (2019) pro-pose different estimators for learning density ratios, either with semi-gradient updates (Sutton, 1988) or by restricting the function class to Reproducing Kernel Hilbert Space.”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, methodology): “Proper estimation of these ratios can improve the accuracy of policy evaluation [21] and the stability of policy learning [12, 14, 22, 40].”
Generalized Off-Policy Actor-Critic (2019, methodology): “= dπ(s) dμ(s) is the density ratio, which is also known as the covariate shift (Gelada and Bellemare, 2019).”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “through a KL divergence penalty [20, 47] or clipping actions [9]), or estimating the learned policy’s stationary distribution and using this estimate when performing an RL update [28, 11, 35].”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, background): “…approaches for policy evaluation that avoid the exponential blow up in importance sampling weights by instead computing a direct ratio over the stationary distribution of state visitations under the target and behavior policy (Hallak and Mannor, 2017; Liu et al., 2018a; Gelada and Bellemare, 2019).”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “Hallak and Mannor [2017] and Gelada and Bellemare [2019] also leverage the same fact in time-invariant MDPs, where they use the stationary ratio of state-action pairs to replace the trajectory weights.”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “Applying GEM-style update to COP-TD (Hallak & Mannor, 2017) is still an open problem as COP-TD involves a nonlinear projection, whose gradient is hard to compute.”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “DICE has been also used to explicitly address the distribution shift in online model-free RL, by directly applying IS on the policy and action-value objectives (Liu et al., 2019; Gelada & Bellemare, 2019).”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, background): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”