Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods

By Ching-An Cheng, Xinyan Yan, Byron Boots. 2019

23 citations recorded by the source corpus.

Policy gradient methods have demonstrated success in reinforcement learning tasks that have high-dimensional continuous state and action spaces. However, policy gradient methods are also notoriously sample inefficient. This can be attributed, at least in part, to the high variance in estimating the gradient of the task objective with Monte Carlo methods. Previous research has endeavored to contend with this problem by studying control variates (CVs) that can reduce the variance of estimates without introducing bias, including the early use of baselines, state dependent CVs, and the more recent state-action dependent CVs. In this work, we analyze the properties and drawbacks of previous CV techniques and, surprisingly, we find that these works have overlooked an important fact that Monte Carlo gradient estimates are generated by trajectories of states and actions. We show that ignoring the correlation across the trajectories can result in suboptimal variance reduction, and we propose a simple fix: a class of "trajectory-wise" CVs, that can further drive down the variance. We show that constructing trajectory-wise CVs can be done recursively and requires only learning state-action value functions like the previous CVs for policy gradient. We further prove that the proposed trajectory-wise CVs are optimal for variance reduction under reasonable assumptions.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Paralleling the development of doubly robust estimators for policy evaluation, doubly robust estimators for the policy gradient have also been derived (Gu et al., 2017a; Huang and Jiang, 2019; Pankov, 2018; Cheng et al., 2019).”
From Importance Sampling to Doubly Robust Policy Gradient (2019, positive): “In fact, the resulting estimator can achieve more variance reduction than Cheng et al. (2019) given additional side information.”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “In the on-policy case, Cheng et al. (2019); Huang and Jiang (2019) propose estimators equivalent to estimating q, d q and plugging into the above equation for ξ NMDP .”