Likelihood ratio gradient estimation for stochastic systems

By Peter W. Glynn. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “What we call the score function estimator (via [3]) is alternatively called the likelihood ratio estimator [5] and REINFORCE [26].”
Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “The score function estimator (SF, also referred to as REINFORCE (Williams, 1992) and likelihood ratio estimator (Glynn, 1990)) uses the identity ∇ θ p θ ( z ) = p θ ( z ) ∇ θ log p θ ( z ) to derive the following unbiased estimator: SF only requires that p θ ( z ) is continuous in θ , and does not…”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “The score function estimator [SFE, 10], also known as the REINFORCE [43] or likelihood-ratio estimator [12], is based on the identity∇φpφ(x) = pφ(x)∇φ log pφ(x), which allows the gradient in Eq.”
Understanding Neural Networks through Representation Erasure (2016, methodology): “The gradient of (5) is approximated using the likelihood ratio trick (Williams, 1992; Glynn, 1990; Aleksandrov et al., 1968), in which for a given e, we sample a sequence of decisions based on π, compute the associated reward and backward propagate gradients to update π, which can be summarized as f”
Concrete Dropout (2017, background): “Several estimators are available for us to do this: for example the score function estimator (also known as a likelihood ratio estimator and Reinforce [6, 12, 30, 35]), or the pathwise derivative estimator (this estimator is also referred to in the literature as the re-parametrisation trick, infinit”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “We derive new ﬂexbile gradient estimators, which combine model-based gradients with the likelihood ratio (LR) trick (Glynn, 1990), also called REINFORCE (Williams, 1992) or the score function estimator.”
Distributed Distributional Deterministic Policy Gradients (2018, methodology): “1.1 R ELATED WORK Historically, estimation of the policy gradient has relied on the likelihood ratio trick (see e.g. Glynn, 1990), more commonly known as REINFORCE (Williams, 1992) in the reinforcement learning community.”
Infinite-Horizon Policy-Gradient Estimation (2001, methodology): “Extensions of the likelihood-ratio method to regenerative processes (including Markov Decision Processes or MDP s) were given by Glynn (1986, 1990), Glynn and L‘Ecuyer (1995) and Reiman and Weiss (1986, 1989), and independently for episodic Partially Observable Markov Decision Processes ( POMDP s)…”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “FORCE rule [139], elsewhere known as the score function [29] or likelihood-ratio estimator [31].”
The Generalized Reparameterization Gradient (2016, methodology): “A general way to obtain unbiased stochastic gradients is to use the score function method, also called log-derivative trick or REINFORCE (Williams, 1992; Glynn, 1990), which has been recently applied to VI (Paisley et al., 2012; Ranganath et al., 2014; Mnih and Gregor, 2014).”
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models (2017, methodology): “Most approaches to optimizing an expectation of a function w.r.t. a discrete distribution based on samples from the distribution can be seen as applications of the REINFORCE (Williams, 1992) gradient estimator, also known as the likelihood ratio (Glynn, 1990) or score-function estimator (Fu, 2006).”
Policy Gradient Methods for Robotics (2006, methodology): “, gPGT = gGMPD, and have been derived previously in the simulation optimization community [27].”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “Because of its widespread and general-purpose nature, it appears under various names throughout the literature, including the score function estimator (Kleijnen and Rubinstein, 1996; Rubinstein et al., 1996), the likelihood ratio method (Glynn, 1990), and the REINFORCE estimator (Williams, 1992).”
Natural actor-critic algorithms (2009, background): “This is the basic idea behind all policy-gradient reinforcement learning methods [70, 48, 61, 42, 11, 53, 3, 35, 21, 22].”
End-to-End Reinforcement Learning of Dialogue Agents for Information Access (2016, methodology): “We can use the likelihood ratio trick (Glynn, 1990) to write the gradient of the objective as follows:”
Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders (2016, background): “Several possibilities for estimating the gradient of discrete variables have been proposed (Glynn, 1990; Titsias & L´azaro-Gredilla, 2015).”