Monte Carlo Gradient Estimation in Machine Learning

By Shakir Mohamed, Mihaela Rosca, Michael Figurnov, Andriy Mnih. 2019

518 citations recorded by the source corpus.

This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a function with respect to parameters defining the distribution that is integrated; the problem of sensitivity analysis. In machine learning research, this gradient problem lies at the core of many learning problems, in supervised, unsupervised and reinforcement learning. We will generally seek to rewrite such gradients in a form that allows for Monte Carlo estimation, allowing them to be easily and efficiently used and analysed. We explore three strategies--the pathwise, score function, and measure-valued gradient estimators--exploring their historical developments, derivation, and underlying assumptions. We describe their use in other fields, show how they are related and can be combined, and expand on their possible generalisations. Wherever Monte Carlo gradient estimators have been derived and deployed in the past, important advances have followed. A deeper and more widely-held understanding of this problem will lead to further advances, and it is these advances that we wish to support.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DADA: Differentiable Automatic Data Augmentation (2020, positive): “In this way, DA optimization becomes a Monte Carlo gradient estimate problem [21].”
BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery (2021, methodology): “This lets us use pathwise gradient estimators [34], which typically have lower variance than the score-function alternatives [60].”
Gradient Estimation with Stochastic Softmax Tricks (2020, background): “Gradients with respect to parameters of the density of an integral are generally intractable, and one must resort to gradient estimators [8, 60].”
Gradients are Not All You Need (2021, result): “A similar set of experiments comparing among gradient estimators has also been shown in Gal [2016], Mohamed et al. [2020].”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “In this work, we make use of the pathwise derivative estimator (Mohamed et al., 2019).”
Learning Discrete Structures for Graph Neural Networks (2019, methodology): “If there exists a differentiable and reversible sampling path sp(θ, ε) for Pθ, with z = sp(θ, ε) for ε ∼ Pε, then one can use the general form of the pathwise gradient estimator (see Mohamed et al. (2019), Sec.”
Training Diffusion Models with Reinforcement Learning (2023, unknown): “The first variant of DDPO, which we call DDPOSF, uses the score function policy gradient estimator, also known as the likelihood ratio method or REINFORCE [53, 29]:”
Pathfinder: Parallel quasi-Newton variational inference (2021, unknown): “This results in a stochastic gradient algorithm, with gradients calculated using automatic differentiation (Mohamed et al., 2019; Carpenter et al., 2015).”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “Low-variance gradient estimates are usually obtained via reparameterization of q(z|x) (Kingma and Welling 2014) although alternatives exist (e.g., inverse CDF sampling) (Mohamed et al. 2020).”
DiBS: Differentiable Bayesian Structure Learning (2021, unknown): “Here, b is a constant with respect to G that can be used for variance reduction [51], and∇Z log p(G |Z) is trivial to compute.”
Differentiation of Blackbox Combinatorial Solvers (2019, methodology): “Accordingly, we will not rely on standard techniques for gradient estimation (see [32] for a comprehensive survey).”