Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning

By Evan Greensmith, Peter L. Bartlett, Jonathan Baxter. 2001

592 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Neural Variational Inference and Learning in Belief Networks (2014, methodology): “The considerable body of work on baselines and other variance reduction methods done in the RL community (e.g. Greensmith et al., 2004) is likely to contain additional techniques relevant for training inference networks.”
High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, background): “In general, the quantity inside the expectation of Equation (6) has lower variance than the quantity inside the expectation of Equation (7), since subtracting the value function acts as a baseline [5].”
Gradient Estimation Using Stochastic Computation Graphs (2015, background): “x [f(x)] generally leads to substantial variance reduction—b is often called a baseline3 (see [6] for a more thorough discussion of baselines and their variance reduction properties).”
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives (2014, methodology): “They thus propose to use a variance reduction approach (see [7] and references therein for example) on the SGD update in order to be able to use constant step sizes and get a linear convergence rate.”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “Variance reduction in policy gradient methods is a long-standing problem with a large body of prior work (Williams, 1992; Greensmith et al., 2004; Schulman et al., 2016).”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, background): “Though the basic version of the estimator can suffer from high variance, various variance reduction techniques can be used to make the estimator much more effective [14].”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “The technique has also been used in machine learning and reinforcement learning to lower variance estimates of gradients (Greensmith et al., 2004; Paisley et al., 2012; Ranganath et al., 2014).”
Transfer Learning with Neural AutoML (2018, methodology): “Subtracting such a baseline is a standard technique in policy gradient algorithms used to reduce the variance of the parameter updates [15].”
Counterfactual reasoning and learning systems: the example of computational advertising (2012, methodology): “t estimation method has been rediscovered multiple times (Aleksandrov et al., 1968; Glynn, 1987; Williams, 1992) in slightly diﬀerent contexts. Variance reduction with a baseline has been analyzed by Greensmith et al. (2002). A.5.5 Notes for section 6 The study of learning principles using uniform c”
Reinforcement learning in robotics: A survey (2013, background): “The resulting update step features the local convergence properties of policy gradient algorithms while reducing update variance (Greensmith et al., 2004).”
Off-policy Bandits with Deficient Support (2020, background): “This connection gives interesting new insight that the baselines used in policy-gradient algorithms not only help to reduce variance in gradients [8], but that they also connect to the problem of support deficiency in the off-policy setting.”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “PG with a baseline (Greensmith et al., 2004) T∑ t=0 ( ∇θ log πtθ ( T∑ t′=t γt ′ rt′ − γtb(st) )) (6) can be derived by taking finite difference over the following OPE estimator, Ĵ(π′) = T∑ t=0 γtρ[0:t] ( rt − b(st) + γb(st+1) ) + b(s0).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, methodology): “2 The most commonly used CVs for policy gradient [1, 17, 18] are state-dependent functions v̂ : S → R, which leads to the difference estimator G̃t := Gt − ( NtV̂t −EAt|St [NtV̂t] ) = Gt −NtV̂t, where V̂t := v̂(St), (4) and the expectation vanishes as EAt|St [NtV̂t] = V̂t∇EAt|St [1] = 0.”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “The LR gradient often has a high variance, and has to be combined with variance reduction techniques known as con-trol variates (Greensmith et al., 2004).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “…[Witten, 1977, Barto et al., 1983] and many techniques since proposed to reduce variance of the estimates of this gradient [Konda and Tsitsiklis, 2000, Weaver and Tao, 2001, Greensmith et al., 2004, Peters et al., 2005, Bhatnagar et al., 2008, 2009, Grondman et al., 2012, Gu et al., 2016].”
Policy Distillation (2015, background): “Also, policies are inherently lower variance than value functions, which should help performance and stability (Greensmith et al., 2004).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “(This follows immediately by applying the corresponding argument in Greensmith et al. (2004) individually to each term in the sum over τ in Equation 2.)”
Learning to Compose Domain-Specific Transformations for Data Augmentation (2017, methodology): “We also use standard techniques of discounting with factor γ ∈ [0, 1] and considering only future rewards [12].”