Annealing between distributions by averaging moments

By Roger Baker Grosse, Chris J. Maddison, Ruslan Salakhutdinov. 2013

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Numerical Analysis, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Sergey Levine
3. Ruslan Salakhutdinov
4. Stefano Ermon
5. Quoc V. Le
6. Mohit Bansal
7. Percy Liang
8. Pieter Abbeel

In Numerical Analysis

1. Benigno Uria
2. Roger Baker Grosse
3. Ruslan Salakhutdinov
4. Frank D. Wood
5. Vaden Masrani
6. Tuan Anh Le
7. Yuri Burda
8. Jascha Narain Sohl-Dickstein

In Statistics and Probability

1. Benigno Uria
2. Tuan Anh Le
3. Ruslan Salakhutdinov
4. Yuri Burda
5. Roger Baker Grosse
6. Surya Ganguli
7. Frank D. Wood
8. Vaden Masrani

In Probabilistic & Bayesian Methods

1. Yoshua Bengio
2. Stefano Ermon
3. Diederik P. Kingma
4. Roger Baker Grosse
5. Ruslan Salakhutdinov
6. Max Welling
7. Frank D. Wood
8. Benigno Uria

Citations in the corpus

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “Figure 5 demonstrates the use of a diffusion model to perform inpainting of a natural image.”
• NICE: Non-linear Independent Components Estimation (2014, background): “addition, the log-likelihood is intractable, and the best known estimation procedure, annealed importance sampling (AIS) (Salakhutdinov and Murray, 2008), might yield an overly optimistic evaluation (Grosse et al., 2013). Directed graphical models lack the conditional independence structure that all”
• Accurate and conservative estimates of MRF log-likelihood using reverse annealing (2014, methodology): “The choice of distributions can have a large effect on the performance of AIS (Grosse et al., 2013), but the most common choice is to take geometric averages of the initial and target distributions: where 0 = β 0 β 1 ... β K = 1 defines the annealing schedule.”
• Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, methodology): “First, we used AIS with moment averages (Grosse et al., 2013), which has been shown to yield more accurate estimates (at a larger computational cost).”
• The Thermodynamic Variational Objective (2019, methodology): “Grosse et al. (2013) studied annealed importance sampling ( AIS ), a related technique that estimates partition functions using a sequence of intermediate distributions using a product of ratios of importance weights.”
• Just-In-Time Learning for Fast and Flexible Inference (2014, background): “[6] study the characteristics of the moment average in detail, and have showed that it can be interpreted as minimizing an objective function mout = argminm U({mout},m) where U({mout},m) = ∑ tDKL(m t out‖m).”
• Connectionist multivariate density-estimation and its application to speech synthesis (2016, methodology): “When Z is intractable, its value can be approximated using Markov chain Monte Carlo techniques like annealed importance sampling (Salakhutdinov and Murray, 2008; Murray and Salakhutdinov, 2009) or moment-averaging annealed importance sampling (Grosse et al., 2013).”