Deep Mixtures of Factor Analysers

By Yichuan Tang, Ruslan Salakhutdinov, Geoffrey E. Hinton. 2012

69 citations recorded by the source corpus.

An efficient way to learn deep density models that have many layers of latent variables is to learn one layer at a time using a model that has only one layer of latent variables. After learning each layer, samples from the posterior distributions for that layer are used as training data for learning the next layer. This approach is commonly used with Restricted Boltzmann Machines, which are undirected graphical models with a single hidden layer, but it can also be used with Mixtures of Factor Analysers (MFAs) which are directed graphical models. In this paper, we present a greedy layer-wise learning algorithm for Deep Mixtures of Factor Analysers (DMFAs). Even though a DMFA can be converted to an equivalent shallow MFA by multiplying together the factor loading matrices at different levels, learning and inference are much more efficient in a DMFA and the sharing of each lower-level factor loading matrix by many different higher level MFAs prevents overfitting. We demonstrate empirically that DMFAs learn better density models than both MFAs and two types of Restricted Boltzmann Machine on a wide variety of datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, methodology): “, Murray, Larochelle In this section, we compare the statistical performance of RNADE to mixtures of Gaussians (MoG) and factor analyzers (MFA), which are surprisingly strong baselines in some tasks (Tang et al., 2012; Zoran and Weiss, 2012). 7.3.1 Low-dimensional data We start by considering three ”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “We compared RNADE to mixtures of Gaussians (MoG) and factor analyzers (MFA), which are surprisingly strong baselines in some tasks [20, 21].”
Masked Autoregressive Flow for Density Estimation (2017, methodology): “Stacking has previously been used in a similar way to improve model fit of deep belief nets [9] and deep mixtures of factor analyzers [28].”
Learning Generative Models with Visual Attention (2013, methodology): “[8] used 24×24 images of centered and cropped faces.”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…1997; Chung et al., 2014), more complex elementary units such as “capsules” and other structures (Delalleau and Bengio, 2011; Hinton et al., 2011; Tang et al., 2012; Livni et al., 2013 ), content addressable (Graves et al., 2014; Weston et al., 2014) and location addressable memories (Graves…”
A RAD approach to deep mixture models (2019, background): “While this is not necessary, we will exploit this structure as highlighted later in the main text and in Figure 4. et al., 2017), some of which can be expressed as a deep mixture model (Tang et al., 2012; Van den Oord & Schrauwen, 2014; van den Oord & Dambre, 2015).”
A tutorial on task-parameterized movement learning and retrieval (2015, background): “As discussed in [92], the prior of each FA can be replaced by a separate second-level MFA that learns to model the aggregated posterior of that FA (instead of the isotropic Gaussian), providing a hierarchical structure organization where one layer of latent variables can be learned at a time.”
Tensor Analyzers (2013, result): “Pixel and GMM results are from (Zoran & Weiss, 2011), while the Deep MFA result is from (Tang et al., 2012).”
Image Modeling with Deep Convolutional Gaussian Mixture Models (2021, methodology): “Previous hierarchical GMM proposals [30]–[32] use (quite complex) extensions of the EM algorithm initialized by kmeans for training.”
Martingale posterior distributions (2021, methodology): “To prevent misleadingly high density values, we remove non-numerical variables and one variable from any pairs with Pearson correlation coefficient greater than 0.98 (e.g. see Tang et al. (2012)).”
Connectionist multivariate density-estimation and its application to speech synthesis (2016, methodology): “If this model is to have a higher marginal likelihood for the training data than the RBM, the new p(h |θ(2)) should better approximate the aggregated posterior (Tang et al., 2012) ∑x pemp(x)p(h |x,θ) (where pemp is the empirical distribution of x) more accurately than p(h |θ) given by the original R”
A Provably Correct Algorithm for Deep Learning that Actually Works (2018, result): “Finally, the works of [13, 14, 16] study generative models similar to ours, with promising empirical results when implementing EM inspired algorithms, but giving no theoretical foundations whatsoever.”
Factoring Variations in Natural Images with Deep Gaussian Mixture Models (2014, methodology): “A closely related method is the deep mixture of factor analyzers (DMFA) model [10], which is an extension of the Mixture of Factor Analyzers (MFA) model [11].”
The student-t mixture as a natural image patch prior with application to image compression (2014, methodology): “Recently similar models to the GMM have been proposed for image modeling, such as the Deep mixture of Factor analyzers (Deep MFA) (Tang et al., 2012).”