Deep Directed Generative Autoencoders

By Sherjil Ozair, Yoshua Bengio. 2014

19 citations recorded by the source corpus.

For discrete data, the likelihood $P(x)$ can be rewritten exactly and parametrized into $P(X = x) = P(X = x | H = f(x)) P(H = f(x))$ if $P(X | H)$ has enough capacity to put no probability mass on any $x'$ for which $f(x')\neq f(x)$, where $f(\cdot)$ is a deterministic discrete function. The log of the first factor gives rise to the log-likelihood reconstruction error of an autoencoder with $f(\cdot)$ as the encoder and $P(X|H)$ as the (probabilistic) decoder. The log of the second term can be seen as a regularizer on the encoded activations $h=f(x)$, e.g., as in sparse autoencoders. Both encoder and decoder can be represented by a deep neural network and trained to maximize the average of the optimal log-likelihood $\log p(x)$. The objective is to learn an encoder $f(\cdot)$ that maps $X$ to $f(X)$ that has a much simpler distribution than $X$ itself, estimated by $P(H)$. This "flattens the manifold" or concentrates probability mass in a smaller number of (relevant) dimensions over which the distribution factorizes. Generating samples from the model is straightforward using ancestral sampling. One challenge is that regular back-propagation cannot be used to obtain the gradient on the parameters of the encoder, but we find that using the straight-through estimator works well here. We also find that although optimizing a single level of such architecture may be difficult, much better results can be obtained by pre-training and stacking them, gradually transforming the data distribution into one that is more easily captured by a simple parametric model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

GSNs : Generative Stochastic Networks (2015, background): “More recently, it was followed up by the various variational auto-encoders or VAEs (Gregor et al., 2014; Kingma & Welling, 2014; Mnih & Gregor, 2014; Rezende et al., 2014), and related directed models (Bornschein & Bengio, 2014; Ozair & Bengio, 2014).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “In [21, 11, 33, 31] variational learning and inference algorithms were developed which allow a ﬂexi-ble generative model and posterior distribution over latent variables to be directly trained against each other.”
NICE: Non-linear Independent Components Estimation (2014, background): “Another interesting comparison is with the work of (Ozair and Bengio, 2014), which also considers a deterministic encoder, but with discrete input and latent variables, which is not always invertible.”
Greedy AutoAugment (2019, background): “Other important usages of data augmentation methods include unsupervised learning (Dosovitskiy et al., 2014; DeVries and Taylor, 2017a; Bengio et al., 2013; Ozair and Bengio, 2014; Naghizadeh et al., 2020b; Naghizadeh and Metaxas, 2020) and in the improved training of generative adversarial networks”
Dataset Augmentation in Feature Space (2017, background): “Bengio et al. (2013) and Ozair & Bengio (2014) claimed that higher level representations expand the relative volume of plausible data points within the feature space, conversely shrinking the space allocated for unlikely data points.”