Black-box $\alpha$-divergence Minimization

By José Miguel Hernández-Lobato, Yingzhen Li, Mark Rowland, Daniel Hernández-Lobato, Thang D. Bui, Richard E. Turner. 2015

152 citations recorded by the source corpus.

Black-box alpha (BB-$\alpha$) is a new approximate inference method based on the minimization of $\alpha$-divergences. BB-$\alpha$ scales to large datasets because it can be implemented using stochastic gradient descent. BB-$\alpha$ can be applied to complex probabilistic models with little effort since it only requires as input the likelihood function and its gradients. These gradients can be easily obtained using automatic differentiation. By changing the divergence parameter $\alpha$, the method is able to interpolate between variational Bayes (VB) ($\alpha \rightarrow 0$) and an algorithm similar to expectation propagation (EP) ($\alpha = 1$). Experiments on probit regression and neural network regression and classification problems show that BB-$\alpha$ with non-standard settings of $\alpha$, such as $\alpha = 0.5$, usually produces better predictions than with $\alpha \rightarrow 0$ (VB) or $\alpha = 1$ (EP).

Subfields: Artificial Intelligence, Control and Systems Engineering, Optimization & Training, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Yarin Gal
  4. Richard E. Turner
  5. Mohit Bansal
  6. Ruslan Salakhutdinov
  7. Pieter Abbeel
  8. Quoc V. Le

In Control and Systems Engineering

  1. Yarin Gal
  2. Lucy R. Chai
  3. Richard E. Turner
  4. Yingzhen Li
  5. José Miguel Hernández-Lobato
  6. Qiang Liu
  7. Finale Doshi-Velez
  8. Zoubin Ghahramani

In Optimization & Training

  1. Lucy R. Chai
  2. Qiang Liu
  3. José Miguel Hernández-Lobato
  4. Yarin Gal
  5. Yingzhen Li
  6. Finale Doshi-Velez
  7. Richard E. Turner
  8. Zoubin Ghahramani

In Information Theory

  1. Qiang Liu
  2. Lucy R. Chai
  3. Finale Doshi-Velez
  4. Yingzhen Li
  5. José Miguel Hernández-Lobato
  6. Richard E. Turner
  7. Cuong V Nguyen
  8. Steffen Udluft

Citations in the corpus