Reward-estimation variance elimination in sequential decision processes
By Sergey Pankov. 2018
5 citations recorded by the source corpus.
Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor sample efficiency. To mitigate this issue, a number of variance-reduction approaches have been proposed. Unfortunately, in the challenging problems with delayed rewards, these approaches either bring a relatively modest improvement or do reduce variance at expense of introducing a bias and undermining convergence. The unbiased methods of gradient estimation, in general, only partially reduce variance, without eliminating it completely even in the limit of exact knowledge of the value functions and problem dynamics, as one might have wished. In this work we propose an unbiased method that does completely eliminate variance under some, commonly encountered, conditions. Of practical interest is the limit of deterministic dynamics and small policy stochasticity. In the case of a quadratic value function, as in linear quadratic Gaussian models, the policy randomness need not be small. We use such a model to analyze performance of the proposed variance-elimination approach and compare it with standard variance-reduction methods. The core idea behind the approach is to use control variates at all future times down the trajectory. We present both a model-based and model-free formulations.
Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Aviral Kumar
- George Tucker
- Chelsea Finn
- Justin Fu
- Ofir Nachum
- Shixiang Shane Gu
- Pieter Abbeel
In Control and Systems Engineering
- Sergey Levine
- Justin Fu
- Aviral Kumar
- George Tucker
- Chelsea Finn
- Byron Boots
- Ching-An Cheng
- Xinyan Yan
In Reinforcement Learning
- Sergey Levine
- Aviral Kumar
- George Tucker
- Justin Fu
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Pieter Abbeel
Citations in the corpus
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Paralleling the development of doubly robust estimators for policy evaluation, doubly robust estimators for the policy gradient have also been derived (Gu et al., 2017a; Huang and Jiang, 2019; Pankov, 2018; Cheng et al., 2019).”
- Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “Upon finishing this work we discovered a recent technical report [24] that is motivated similarly and details exactly the equation (11) that TrajCV uses.”