A Tutorial on Bayesian Nonparametric Models

By Samuel J. Gershman, David M. Blei. 2011

644 citations recorded by the source corpus.

A key problem in statistical modeling is model selection, that is, how to choose a model at an appropriate level of complexity. This problem appears in many settings, most prominently in choosing the number of clustersinmixturemodelsorthenumberoffactorsinfactoranalysis.Inthistutorial,wedescribeBayesian nonparametric methods, a class of methods that side-steps this issue by allowing the data to determine the complexity of the model. This tutorial is a high-level introduction to Bayesian nonparametric methods and contains several examples of their application.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic variational inference (2012, methodology): “For a reviews of BNP methods, see the edited volume of Hjort et al. (2010) and the tutorial of Gershman and Blei (2012).”
Hierarchical Variational Models (2015, background): “Mixtures can approximate arbitrary distributions (given enough components), and have been considered as variational families (Jaakkola and Jordan, 1998; Lawrence, 2000; Gershman and Blei, 2012; Salimans et al., 2013).”
Recent Advances in Open Set Recognition: A Survey (2018, background): “Dirichlet process (DP) [120], [121], [122], [123], [124] considered as a distribution over distributions is a stochastic process, which has been widely applied in clustering and density estimation problems as a nonparametric prior defined over the number of mixture components.”
Mining Semantic Loop Idioms from Big Code (2016, background): “Bayesian nonparametrics have been the subject of intense research in statistics and machine learning (Gershman and Blei 2012; Teh and Jordan 2010).”
Reinforcement Learning and Causal Models (2017, background): “…appealing to ideas from Bayesian nonparametrics, a field of statistics that deals with inference over latent structures with unbounded 4 More precisely, ramps will occur when the value function is a convex function of the state representation (Ger- shman, 2014). complexity (Gershman & Blei, 2012).”
Reconciling meta-learning and continual learning with online mixtures of tasks (2018, methodology): “The most widely used is the Gibbs sampler [35, 17], which draws from the conditional distribution of each latent variable, given the others, until convergence to the posterior distribution over all the latents.”
Online gradient-based mixtures for transfer modulation in meta-learning (2018, methodology): “The most widely used is the Gibbs sampler (Neal, 2000; Gershman and Blei, 2012), which draws from the condi- tional distribution of each latent variable given the others until convergence to the posterior distribution over all the latents.”
Reinforcement learning improves behaviour from evaluative feedback (2015, positive): “One highly desirable property, however, is that it provides a direct way of applying the emerging class of non-parametric Bayesian methods 12 to decision making — if you can model a process probabilistically, you can use that model to make a selection.”
Variational Russian Roulette for Deep Bayesian Nonparametrics (2019, methodology): “A principled method from statistics to choose the complexity of a model is provided by Bayesian nonparametric methods (Walker et al., 1999; Müller & Quintana, 2004; Orbanz & Teh, 2010; Gershman & Blei, 2012).”
Time series forecasting for nonlinear and non-stationary processes: a review and comparative study (2015, methodology): “, estimating the conditional distribution p(θ |y) of the hidden model or parameters θ given an observed time series y(t) (Gershman and Blei, 2012).”
Novelty or Surprise? (2013, background): “Of current interest in statistics and machine learning are Bayesian non-parametric clustering methods (Gershman and Blei, 2012).”
Mining idioms from source code (2014, methodology): “Nonparametric Bayesian methods have become enormously popular in statistics, machine learning, and natural language processing because they provide a flexible and principled way of automatically inducing a “sweet spot” of model complexity based on the amount of data that is available [41, 16, 48].”
Discovering latent causes in reinforcement learning (2015, methodology): “This combination of simplicity and flexibility can be formalized using concepts from Bayesian nonparametric statistics [19].”
Individual heterogeneity and capture–recapture models: what, why and how? (2018, methodology): “Rather than comparing models that vary in complexity using for instance an information criterion, Bayesian nonparametric approaches fit a single model that can adapt its complexity to the data (Gershman and Blei 2012; see Ford et al. 2015 and Manrique-Vallier 2016 for applications to CR models).”