Learning the Structure of Deep Sparse Graphical Models

By Ryan P. Adams, Hanna M. Wallach, Zoubin Ghahramani. 2009

90 citations recorded by the source corpus.

Deep belief networks are a powerful way to model complex probability distributions. However, it is difficult to learn the structure of a belief network, particularly one with hidden units. The Indian buffet process has been used as a nonparametric Bayesian prior on the structure of a directed belief network with a single infinitely wide hidden layer. Here, we introduce the cascading Indian buffet process (CIBP), which provides a prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network framework to allow each unit to vary its behavior between discrete and continuous representations. We use Markov chain Monte Carlo for inference in this model and explore the structures learned on image data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “[32] proposed a nonparametric Bayesian approach that searches over an infinite network using MCMC.”
Connectivity Learning in Multi-Branch Networks (2017, methodology): “n the network, thus optimizing over a much larger family of connectivity models. While our algorithm is limited to optimizing the connectivity structure within a predeﬁned architecture, Adams et al. (Adams et al., 2010) proposed a nonparametric Bayesian approach that searches over an inﬁnite network”
High-Dimensional Probability Estimation with Deep Density Models (2013, background): “Directed belief networks [Pearl, 1988, Neal, 1992, Adams et al., 2010], on the other hand, enable one to specify a priori marginals of hidden variables and are easily normalized, but require costly inference procedures.”
Sum-product networks: A new deep architecture (2011, background): “Some recent machine learning works also showed selected image completion results [16, 1, 30], but they were limited and often focused on small images.”
Online Incremental Feature Learning with Denoising Autoencoders (2012, methodology): “Alternatively, there are several methods for jointly training the feature mapping and the classiﬁer, such as via multi-task learning [31, 6], transfer learning [2, 22, 17], non-parametric Bayesian learning [1, 10], and deep learning [16, 3, 28, 27, 20].”
Gen: a general-purpose probabilistic programming system with programmable inference (2019, methodology): “MCMC for Bayesian Structure Learning One approach to inferring the structure of a model from data involves placing a prior distribution on the set of possible structures and jointly inferring the structure and the numerical parameters using MCMC for posterior inference [2, 38, 67].”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, methodology): “In [1] the authors use a cascaded Indian buﬀet process to learn the structure of sigmoidal belief networks.”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, methodology): “Doing so connects to older work on Bayesian marginalization over structures [37, 1].”
Beyond Manual Tuning of Hyperparameters (2015, background): “For instance, some network architectures can be treated using hyper-priors [1] while the setting of others is done manually or left to hyperparameter optimization methods [8, 64].”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, background): “In related but orthogonal research (Adams et al., 2010; Song et al., 2017) focused on the problem of structure learning in deep belief networks.”
Probabilistic machine learning and artificial intelligence (2015, methodology): “The Indian buffet process (IBP) 40 is a non-parametric model that can be used for latent feature modelling, learning overlapping clusters, sparse matrix factorization, or to non-parametrically learn the structure of a deep network 41 .”
Automatic model construction with Gaussian processes (2014, background): “For example, deep generative models (Adams et al., 2010; Bengio et al., 2013; Damianou and Lawrence, 2013; Rippel and Adams, 2013; Salakhutdinov and Hinton, 2009) could be considered an example of the bottom-up approach, since they compose individual model “layers” to produce more powerful models.”
Learning the Structure of Deep Convolutional Networks (2015, methodology): “Fortunately, model structure learning has been elegantly explored by several generative model priors, such as the Indian Buffet Process (IBP) [9, 1] or Beta Process (BP) [21].”
Selecting Receptive Fields in Deep Networks (2011, methodology): “[1] use a non-parametric Bayesian prior to jointly infer the depth and number of hidden units at each layer of a deep belief network during training.”
Variational Russian Roulette for Deep Bayesian Nonparametrics (2019, methodology): “Within machine learning, Bayesian nonparametric methods have been applied within models as diverse as clustering (Antoniak, 1974; Görür & Rasmussen, 2010; Teh et al., 2005), topic modeling (Teh et al., 2006), and infinite deep neural networks (Adams et al., 2010).”