Likelilood ratio gradient estimation: an overview

By Peter W. Glynn. 1987

The likelihood ratio method for gradient estimation is briefly surveyed. Two applications settings are described, namely Monte Carlo optimization and statistical analysis of complex stochastic systems. Steady-state gradient estimation is emphasized, and both regenerative and non-regenerative approaches are given. The paper also indicates how these methods apply to general discrete-event simulations; the idea is to view such systems as general state space Markov chains.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Counterfactual reasoning and learning systems: the example of computational advertising (2012, methodology): “any predictor whose conditional expectation ζ∗(υ) = R ω\υ ζ(ω)P ∗(ω|υ) can be computed eﬃciently. The policy gradient estimation method has been rediscovered multiple times (Aleksandrov et al., 1968; Glynn, 1987; Williams, 1992) in slightly diﬀerent contexts. Variance reduction with a baseline has b”
Reinforcement learning in robotics: A survey (2013, background): “Note, that this approach is well-known in the simulation community (Glynn, 1987) with fixed models.”
Independently Controllable Factors (2017, methodology): “In our experiments, the gradient on line 6 is also computed by backpropagation and sampling of the expectation, while the gradient on line 7 is computed with the REINFORCE (Glynn, 1987; Williams, 1992) estimator: where b ( s ) is a baseline function, which can for example be chosen to be the mean re”
Model learning for robot control: a survey (2011, unknown): “, by resetting the random seed, a trick well-known in simulation optimization [44] which is the most common way to implement PEGASUS for complex systems [104]).”
On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient (2010, methodology): “REINFORCE has been shown to be moderately efficient in terms of number of samples used [6, 7].”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “Glynn and Szechtman (2002) describe the connections between control variates and other variance reduction approaches such as antithetic and stratified sampling, conditional Monte Carlo and non-parametric maximum likelihood.”
Unsupervised Recurrent Neural Network Grammars (2019, methodology): “9 An estimator for the gradient with respect to Eqφ(z |x)[log pθ(x, z)] is obtained via the score function gradient estimator (Glynn, 1987; Williams, 1992),”
Policy Gradient in Continuous Time (2006, methodology): “(Reiman and Weiss, 1986; Glynn, 1987) or more recently in the reinforcement learning literature (Williams, 1992; Sutton et al.”
Online Model Learning Algorithms for Actor-Critic Control (2015, methodology): “by using infinitesimal perturbation analysis (IPA) or likelihood-ratio methods (Aleksandrov et al., 1968; Glynn, 1987).”
Policy Gradient Methods for Robotics (2006, methodology): “This type of method is guaranteed to converge to the true gradient at the fastest theoretically possible pace of O ( I−1/2 ) where I denotes the number of roll-outs [22] even if the data is generated from a highly stochastic system.”
Adversarially Regularized Autoencoders (2017, methodology): “ider applying GANs to discrete data such as text. Policy gradient methods are a natural way to deal with the resulting non-differentiable generator objective when training directly in discrete space (Glynn, 1987; Williams, 1992). When trained on text data however, such methods often require pre-trai”
Monte Carlo and Quasi-Monte Carlo Sampling (2009, methodology): “Alternative approaches are infinitesimal perturbation analysis (IPA— also called pathwise differentiation) [133, 144, 150, 192] and the likelihood ratio method (LR — also called the score function method) [133, 152].”