Better Mixing via Deep Representations

By Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, Salah Rifai. 2012

356 citations recorded by the source corpus.

It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Adversarial Nets (2014, background): “The σ parameter Model MNIST TFD DBN [3] 138 ± 2 1909 ± 66 Stacked CAE [3] 121 ± 1 .”
Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “More recently, Bengio et al. (2013c) generalized this to arbitrary variables (discrete, continuous or both), arbitrary corruption (not necessarily asymptotically small), and arbitrary loss function (so long as they can be seen as a log-likelihood).”
Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “at test examples) but high in general. Different variants of auto-encoders and sparse coding have been, along with RBMs, among the most successful building blocks in recent research in deep learning (Bengio et al., 2013b). Whereas the usefulness of auto-encoder variants as feature learners for super”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “One solution to the problem of slow-sampling Markov Chains is to perform sampling using a carfefully crafted latent space (Bengio et al., 2013; Hoffman et al., 2019).”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “VW is similar in spirit to DAE (Bengio et al., 2013b), and NET approaches (Sohl-Dickstein et al., 2015) but it retains two crucial differences.”
Learning to Generate Samples from Noise through Infusion Training (2017, neutral): “Model Test DBM (Bengio et al., 2013) 138 ± 2 SCAE (Bengio et al., 2013) 121 ± 1 .”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, neutral): “6 bits DBN [2] 138 ± 2 bits Deep GSN [3] 214 ± 1 .”
NICE: Non-linear Independent Components Estimation (2014, methodology): “In this paper, we can observe such ﬂattening as found in (Bengio et al., 2013) with stacked autoencoders and RBMs.”
Generative Adversarial Text to Image Synthesis (2016, background): “Deep networks have been shown to learn representations in which interpolations between embedding pairs tend to be near the data manifold (Bengio et al., 2013; Reed et al., 2014).”
Conditional Generative Adversarial Nets (2014, background): “Model MNIST DBN [1] 138± 2 Stacked CAE [1] 121± 1.”
Generative adversarial networks (2014, neutral): “ractable for all but the most trivial instances, although they can be estimated by Markov chain Monte Carlo (MCMC) methods. Mixing poses a signiﬁcant problem for learning algorithms that rely on MCMC [3, 5]. Deep belief networks (DBNs) [16] are hybrid models containing a single undirected layer and ”
Conditional Image Synthesis with Auxiliary Classifier GANs (2016, background): “In an overﬁt model one might observe discrete transitions in the interpolated images and regions in latent space that do not correspond to meaningful images (Bengio et al., 2012; Radford et al., 2015; Dinh et al., 2016).”
Adversarial Autoencoders (2015, methodology): “To determine whether the model is over-ﬁtting by copying the training data points, we used the last column of these MNIST (10K) MNIST (10M) TFD (10K) TFD (10M) DBN [Hinton et al., 2006] 138 ± 2 - 1909 ± 66 - Stacked CAE [Bengio et al., 2013] 121 ± 1 .”
A note on the evaluation of generative models (2015, methodology): “Following Bengio et al. (2013a), Parzen windows are in practice commonly ﬁt to only 10,000 samples (e.g., Bengio et al., 2013b; Goodfellow et al., 2014; Li et al., 2015; Sohl-Dickstein et al., 2015).”
Generative Moment Matching Networks (2015, methodology): “This concept is referred to in the literature as disen-tangling factors of variation (Bengio et al., 2013a).”
Representation Learning: A Review and New Perspectives (2012, background): “1" Bengio et al. (2013) suggest that deep representations could help mixing between such well separated modes, based on both theoretical arguments and on empirical evidence.”
Representation Learning by Learning to Count (2017, background): “We make the hypothesis that the model needs to disentangle the image into high-level factors of variation, such that the complex relation between the original image and its regions is translated to a simple arithmetic operation [3, 35].”
Demystifying MMD GANs (2018, background): “Once the images are mapped through appropriately trained convolutional layers, however, they can follow a much simpler distribution with broader support across the mapped domain: a phenomenon also observed in autoen-coders (Bengio et al., 2013).”
How to Construct Deep Recurrent Neural Networks (2013, background): “Previous work has shown that higher-level representations of deep networks tend to better disentangle the underlying factors of variation than the original input (Goodfellow et al. , 2009; Glorot et al. , 2011b) and ﬂatten the manifolds near which the data concentrate (Bengio et al. , 2013).”