Variational Tempering

By Stephan Mandt, James McInerney, Farhan Abrol, Rajesh Ranganath, David M. Blei. 2014

61 citations recorded by the source corpus.

Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “Some of them require extra time to collect necessary statistics such as the gradient magnitude of each sample [12, 1], change the network architecture [15, 44], or involve an annealing schedule like selfpaced learning [25, 32].”
Factorized Variational Autoencoders for Modeling Audience Reactions to Movies (2017, background): “[14] proposed to disentangle the latent dimensions by upweighting the KullbackLeibler (KL) divergence between prior and variational distribution which relates to variational tempering [31].”
Balanced Mini-batch Sampling for SGD Using Determinantal Point Processes (2017, background): “In the future, we will explore the possibility to further improve the efﬁciency of the algorithm with data reweighing [28] and tackle imbalanced learning problems involving different modalities for supervised [47] and multi-modal [15] settings.”
Stochastic Learning on Imbalanced Data: Determinantal Point Processes for Mini-batch Diversification (2017, background): “In the future, we will explore the possibility to further improve the efficiency of the algorithm with data reweighing [27] and tackle imbalanced learning problems involving different modalities for supervised [46] and multi-modal [15] settings.”
Deep Probabilistic Video Compression (2018, background): “This is achieved by the β-VAE loss (Higgins et al., 2016; Mandt et al., 2016), which takes the form (up to constant terms) − Ef̃ ,z̃1:T∼q[log pθ(x1:T |f̃ , z̃1:T )]− β Ef̃ ,z̃1:T∼q[log pθ(f̃ , z̃1:T )] = Ef̃ ,z̃1:T∼q T∑ t=1 ‖x̃t − xt‖1 + βH [ qφ(z̃1:T , f̃ | x1:T ), pθ(f̃ , z̃1:T ) ] (5) where the…”
Boosting Variational Inference: an Optimization Perspective (2017, background): “On the other hand, finding these individual components is non-convex and is known to exhibit several local optima [19, 1].”