From Importance Sampling to Doubly Robust Policy Gradient

By Jiawei Huang, Nan Jiang. 2019

26 citations recorded by the source corpus.

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provide a simple derivation of a very general and flexible form of PG, which subsumes the state-of-the-art variance reduction technique (Cheng et al., 2019) as its special case and immediately hints at further variance reduction opportunities overlooked by existing literature. We analyze the variance of the new DR-PG estimator, compare it to existing methods as well as the Cramer-Rao lower bound of policy gradient, and empirically show its effectiveness.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Networks and Communications

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Paralleling the development of doubly robust estimators for policy evaluation, doubly robust estimators for the policy gradient have also been derived (Gu et al., 2017a; Huang and Jiang, 2019; Pankov, 2018; Cheng et al., 2019).”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “In the on-policy case, Cheng et al. (2019); Huang and Jiang (2019) propose estimators equivalent to estimating q, d q and plugging into the above equation for ξ NMDP .”