Nonlinear Models Using Dirichlet Process Mixtures

By Babak Shahbaba, Radford M. Neal. 2007

297 citations recorded by the source corpus.

We introduce a new nonlinear model for classification, in which we model the joint distribution of response variable, y, and covariates, x, non-parametrically using Dirichlet process mixtures. We keep the relationship between y and x linear within each component of the mixture. The overall relationship becomes nonlinear if the mixture contains more than one component, with different regression coefficients. We use simulated data to compare the performance of this new approach to alternative methods such as multinomial logit (MNL) models, decision trees, and support vector machines. We also evaluate our approach on two classification problems: identifying the folding class of protein sequences and detecting Parkinson's disease. Our model can sometimes improve predictive accuracy. Moreover, by grouping observations into sub-populations (i.e., mixture components), our model can sometimes provide insight into hidden structure in the data.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “Since then, many different MoE variants emerged, including mixture of SVM models (Collobert et al., 2002), Dirichlet processes (Shahbaba & Neal, 2009) and various neural networks.”
A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning (2020, methodology): “The joint distribution modeled by each component can be decomposed as p(x, y|z) = p(y|x, z)p(x|z) (Rasmussen & Ghahramani, 2002; Shahbaba & Neal, 2009).”
A Tutorial on Bayesian Nonparametric Models (2011, methodology): “Recently, another nonparametric approach to supervised learning has been developed, based on the CRP mixture model (Shahbaba and Neal, 2009; Hannah et al., 2010).”
Probabilistic multi-class multi-kernel learning: on protein fold recognition and remote homology detection (2008, background): “The need for such information to tackle the two misclassification patterns can also be seen in the work of (Shahbaba and Neal, 2007) where even when the problem is treated as a hierarchical classification with parent classes, the performance is not improving beyond 61.4%.”
Bayesian Nonparametric Modeling for Multivariate Ordinal Regression (2014, methodology): “Shahbaba and Neal (2009) considered classification of a univariate response using a multinomial logit kernel, and this was extended by Hannah, Blei, and Powell (2011) to accommodate alternative response types with mixtures of generalized linear models.”
Residual Mixture of Experts (2022, unknown): “Various of expert architectures are proposed [10, 12, 43, 48, 51].”
Novel Speech Signal Processing Algorithms for High-Accuracy Classification of Parkinson's Disease (2012, unknown): “…50 replicates 91.4 ± 4.4 Psorakis et al. 2010 [41] Non-sparse E-M, 10-fold crossvalidation with 10 repetitions 89.5 ± 6.6 Shahbaba and Neal 2009 [42] dpMNL, 5-fold cross-validation 87.7 ± 3.3 *Optimal 4 feature subset from Little et al. 2009 [5] SVM methodology in this study, 10-fold…”
Martingale posterior distributions (2021, background): “Examples of this are Müller et al. (1996); Shahbaba and Neal (2009); Hannah et al. (2011), where the prior on f0(yi,xi) is a DPMM.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Restrictions apply. been extensively employed in a range of early proposals [15], [16], [17], [18], [19], [20], [21], [22], [23], [59].”