Exploiting compositionality to explore a large space of model structures

By Roger Baker Grosse, Ruslan Salakhutdinov, William T. Freeman, Josh Tenenbaum. 2012

113 citations recorded by the source corpus.

The recent proliferation of richly structured probabilistic models raises the question of how to automatically determine an appropriate model for a dataset. We investigate this question for a space of matrix decomposition models which can express a variety of widely used models from unsupervised learning. To enable model selection, we organize these models into a context-free grammar which generates a wide variety of structures through the compositional application of a few simple rules. We use our grammar to generically and efficiently infer latent components and estimate predictive likelihood for nearly 2500 structures using a small toolbox of reusable algorithms. Using a greedy search over our grammar, we automatically choose the decomposition structure from raw data by evaluating only a small fraction of all models. The proposed method typically finds the correct structure for synthetic data and backs off gracefully to simpler models under heavy noise. It learns sensible structures for datasets as diverse as image patches, motion capture, 20 Questions, and U.S. Senate votes, all using exactly the same code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Neural Architecture Search & AutoML

Citations in the corpus

Progressive Neural Architecture Search (2017, background): “Other related works include [22], who focus on MLP rather than CNNs; [17], who used an incremental approach in the context of evolutionary algorithms; [5] who used a schedule of increasing number of layers; and [23] who search through the space of latent factor models specified by a grammar.”
Neural Architecture Optimization (2018, background): “Some recent works [17, 19, 27] try to improve the efficiency in architecture search by exploring the search space incrementally and sequentially, typically from shallow to hard.”
Hidden physics models: Machine learning of nonlinear partial differential equations (2017, background): “For a more systematic treatment of the kernelselection problem we would like to refer the readers to [17, 18, 19].”
Automatic Construction and Natural-Language Description of Nonparametric Regression Models (2014, positive): “Searching over open-ended model spaces This work was inspired by previous successes at searching over open-ended model spaces: matrix decompositions (Grosse, Salakhutdinov, and Tenenbaum, 2012) and graph structures (Kemp and Tenenbaum, 2008).”
Venture: a higher-order probabilistic programming platform with programmable inference (2014, background): “…of graphical models (Heckerman, 1998; Friedman and Koller, 2003; Mansinghka et al., 2006), models for inductive learning of symbolic expressions (Grosse et al., 2012; Duvenaud et al., 2013) and models defined in terms of complex stochastic simulation software that lack tractable likelihoods…”
Probabilistic machine learning and artificial intelligence (2015, unknown): “an open-ended language of models 82 .”
Automatic Sampler Discovery via Probabilistic Programming and Approximate Bayesian Computation (2016, background): “The latter, generalising from data, is usually referred to as either density estimation or learning, and also includes automatic modelling [10,7].”
UvA-DARE (Digital Academic Repository) Neural Relational Inference for Interacting Systems Neural Relational Inference for Interacting Systems (2018, unknown): “, 2016), it is difficult to find the correct structure as the search space is very large (Grosse et al., 2012).”
Identifying Causal Structure in Large-Scale Kinetic Systems (2018, methodology): “…approaches tackle the problem by using model-free network reconstructions as in Äijö and Lähdesmäki [2009], Guo et al. [2014], Casadiego et al. [2017] and also methods based on general time-series models, which focus on non-parametric kernel methods [Grosse et al., 2012, Duvenaud et al., 2013].”
Applications of Probabilistic Programming (Master's thesis, 2015) (2016, background): “Also worthwhile of mentioning are recent works in search over generative probabilistic model structures (Grosse et al., 2012) and kernel compositions (Duvenaud et al., 2013).”
Using Regression Kernels to Forecast A Failure to Appear in Court (2014, background): “10There is interesting work that attempts to open the black box (e.g., Grosse et al., 2012; Duvenaud et al., 2013) so that the ways in which linear basis expansions are captured in K are better understood.”
Gen: a general-purpose probabilistic programming system with programmable inference (2019, methodology): “Recurse is used to implement recursive computation patterns including probabilistic context free grammars, which are a common building block of probabilistic models [17, 25, 33].”
Structure Discovery in Nonparametric Regression through Compositional Kernel Search (2013, methodology): “Borrowing discrete search techniques which have proved successful in equation discovery (Todorovski & Dzeroski, 1997) and unsupervised learning (Grosse et al., 2012), we automatically search over this space of kernel structures using marginal likelihood as the search criterion.”
Sandwiching the marginal likelihood using bidirectional Monte Carlo (2015, methodology): “In the grammar of Grosse et al. (2012), this model would be written as GG + G .”