Bidirectional Helmholtz Machines

By Jörg Bornschein, Samira Shabanian, Asja Fischer, Yoshua Bengio. 2015

26 citations recorded by the source corpus.

Efficient unsupervised training and inference in deep generative models remains a challenging problem. One basic approach, called Helmholtz machine, involves training a top-down directed generative model together with a bottom-up auxiliary model used for approximate inference. Recent results indicate that better generative models can be obtained with better approximate inference procedures. Instead of improving the inference procedure, we here propose a new model which guarantees that the top-down and bottom-up distributions can efficiently invert each other. We achieve this by interpreting both the top-down and the bottom-up directed models as approximate inference distributions and by defining the model distribution to be the geometric mean of these two. We present a lower-bound for the likelihood of this model and we show that optimizing this bound regularizes the model so that the Bhattacharyya distance between the bottom-up and top-down approximate distributions is minimized. This approach results in state of the art generative models which prefer significantly deeper architectures while it allows for orders of magnitude more efficient approximate inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Unrolled Generative Adversarial Networks (2016, background): “…et al., 2015), autoregressive models (Theis & Bethge, 2015; van den Oord et al., 2016a;b), real non-volume preserving transformations (Dinh et al., 2014; 2016), Helmholtz machines (Dayan et al., 1995; Bornschein et al., 2015), and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).”
Ladder Variational Autoencoders (2016, methodology): “To improve performance we fine-tuned the best performing five layer LVAE models by training these for a further 2000 epochs with annealed learning rate and increasing the number of IW samples and see a slight improvements in the test set log-likelihood values, Table 1.”
Discrete Variational Autoencoders (2016, background): “…variables — as do DBMs (Salakhutdinov & Hinton, 2009), NADEs (Larochelle & Murray, 2011), sigmoid belief networks (Spiegelhalter & Lau-ritzen, 1990; Bornschein et al., 2016), and DARNs (Gregor et al., 2014) — or exclusively continuous variables — as do VAEs (Kingma & Welling, 2014; Rezende et al.,…”
The Thermodynamic Variational Objective (2019, background): “[33], who introduced another way of improving deep generative modeling through geometrically interpolating between distributions and using importance sampling to estimate gradients.”
An Introduction to Variational Autoencoders (2019, unknown): “, 2016) and Bi-directional Helmholtz Machines (Bornschein et al., 2016).”
Undirected Graphical Models as Approximate Posteriors (2019, background): “…grouped into three categories: i) Training objectives that replace the variational bound with tighter bounds (Burda et al., 2015; Li & Turner, 2016; Bornschein et al., 2016). ii) Autoregressive models (Hochreiter & Schmidhuber, 1997; Graves, 2013) that use DGMs to form more flexible distributions…”