Variational inference for Dirichlet process mixtures

By Michael I. Jordan. 2006

1,625 citations recorded by the source corpus.

Dirichlet process (DP) mixture models are the cornerstone of non- parametric Bayesian statistics, and the development of Monte-Carlo Markov chain (MCMC) sampling methods for DP mixtures has enabled the application of non- parametric Bayesian methods to a variety of practical data analysis problems. However, MCMC sampling can be prohibitively slow, and it is important to ex- plore alternatives. One class of alternatives is provided by variational methods, a class of deterministic algorithms that convert inference problems into optimization problems (Opper and Saad 2001; Wainwright and Jordan 2003). Thus far, varia- tional methods have mainly been explored in the parametric setting, in particular within the formalism of the exponential family (Attias 2000; Ghahramani and Beal 2001; Blei et al. 2003). In this paper, we present a variational inference algorithm for DP mixtures. We present experiments that compare the algorithm to Gibbs sampling algorithms for DP mixtures of Gaussians and present an application to a large-scale image analysis problem.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “…research have shown that variational inference does not necessarily suffer in accuracy, for example, in terms of posterior predictive densities (Blei and Jordan 2006; Braun and McAuliffe 2010; Kucukelbir et al. 2017); other research focuses on where variational inference falls short,…”
Kernel Bayes' Rule (2010, methodology): “n the form of a processwithout a density form. In this case, sampling methods are often applied ([21, 22, 41] among others). Alternatively, the posterior may be approximated using variational methods [4]. We will present an experimental comparison of KBR and ABC in Section 5.2. • Even if explicit fo”
A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning (2020, methodology): “Since the exact inference of the posterior of DPM models is infeasible, approximate inference methods are adopted such as Markov chain Monte Carlo (MCMC) (Maceachern, 1994; Escobar & West, 1995; Neal, 2000) or variational inference (Blei & Jordan, 2006; Wang & Dunson, 2011; Lin, 2013).”
Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables (2017, background): “Although apparently arbitrary, our interest here is justified by the fact that this quantity naturally arises in the context of (variational) Bayesian inference on Dirichlet processes (Blei and Jordan, 2005), which are quite popular in the context of non-parametric data clustering problems.”
Stick-Breaking Variational Autoencoders (2016, background): “As explained by Blei & Jordan (2006), the truncation level is a variational parameter and not part of the prior model speciﬁcation.”
Dirichlet Process Mixtures of Generalized Linear Models (2009, methodology): “ith the DP-GLM is computational eciency as implemented. All results were generated using MCMC, which does not scale well to large datasets. An alternative implementation using variational inference (Blei and Jordan 2005), possibly online variational inference (Sato 2001), would greatly increase comp”
Coresets for Nonparametric Estimation - the Case of DP-Means (2015, methodology): “In the case of Dirichlet process mixtures the most popular approaches include recursive approximations (Newton & Zhang, 1999), sequential importance sampling (MacEachern et al., 1999) and variational Bayes (Blei & Jordan, 2006).”
MXFusion: A Modular Deep Probabilistic Programming Library (2018, methodology): “This allows us to package a sophisticated probabilistic model with their proposed inference methods as a re-usable building block, e.g., Dirichlet process with variational inference (Blei & Jordan, 2006).”
Visualization and analysis of single-cell RNA-seq data by kernel-based similarity learning (2016, methodology): “We found that K-means clustering with SIMLR consistently outperforms the existing alternatives: Non-Negative Matrix Factorization (NMF) [34], Gaussian Mixture Models (GMM) [35], Dirichlet Process Mixture Models (DPMM) [36], on the four data sets (Table 4).”
Multi-Task Learning for Classification with Dirichlet Process Priors (2007, methodology): “Following Blei and Jordan (2005), we employ a truncated stick-breaking representation for the variational distribution. It is worth noting that in the VB framework, the model is a non-truncated full Dirichlet process while the variational distribution is truncated. Empirical results on truncation le”
Fast Bayesian Inference in Dirichlet Process Mixture Models (2011, methodology): “We found the Blei and Jordan (2006) approach to have better performance than the newer VB variants in simulations (results not shown).”
Optimism-driven exploration for nonlinear systems (2015, methodology): “efficiency of our algorithm also improves on the Gaussian process used by PILCO, since with a suitable online learning method, the complexity of training the DPMM grows as O(n log n) in the number of data points [8], and could be improved to be sublinear with an online algorithm [11].”
The Infinite PCFG Using Hierarchical Dirichlet Processes (2007, methodology): “However, we chose to use variational inference (Blei and Jordan, 2005), which provides a fast deterministic alternative to sampling, hence avoiding issues of diagnosing convergence and aggregating samples.”
Fast unsupervised ego-action learning for first-person sports videos (2011, methodology): “Typical inference algorithm include MCMC [5], variational inference [1], multi-pass online inference [15] and online beam search [2].”
Time series forecasting for nonlinear and non-stationary processes: a review and comparative study (2015, background): “Additionally, Dirichlet Process (DP) nonparametric models (Blei and Jordan, 2005) are being investigated for adaptively determining the number of clusters based on the complexity of the data.”
Variational Denoising Network: Toward Blind Noise Modeling and Removal (2019, methodology): “For any noisy image y and its simulated “clean” image x in the training set, we can decompose its marginal likelihood as the following form [7]: log p(y|z,σ(2)) = L(z,σ(2);y) +DKL ( q(z,σ(2)|y)||p(z,σ(2)|y) ) , (6)”