Bayesian Hierarchical Mixtures of Experts

By Charles M. Bishop, Markus Svensén. 2002

172 citations recorded by the source corpus.

The Hierarchical Mixture of Experts (HME) is a well-known tree-structured model for regression and classification, based on soft probabilistic splits of the input space. In its original formulation its parameters are determined by maximum likelihood, which is prone to severe overfitting, including singularities in the likelihood function. Furthermore the maximum likelihood framework offers no natural metric for optimizing the complexity and structure of the tree. Previous attempts to provide a Bayesian treatment of the HME model have relied either on local Gaussian representations based on the Laplace approximation, or have modified the model so that it represents the joint distribution of both input and output variables, which can be wasteful of resources if the goal is prediction. In this paper we describe a fully Bayesian treatment of the original HME model based on variational inference. By combining 'local' and 'global' variational methods we obtain a rigorous lower bound on the marginal probability of the data under the model. This bound is optimized during the training phase, and its resulting value can be used for model order selection. We present results using this approach for data sets describing robot arm kinematics.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “2019) and has been studied in multi-task (for example (Bishop and Svensén 2012; Lin et al. 2019)) and metalearning (for example, (Rajendran, Irpan, and Jang 2020)).”
Causal Falling Rule Lists (2015, neutral): “Thus, the model selection process is similar to that of Bayesian Hierarchical Mixture of Experts (Bishop and Svenskn 2002), but with hard partitioning of data and dependence between experts.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “6 is extended to a hierarchical architecture [125]; the Bayesian approach is employed instead of the maximum likelihood to get rid of overfitting and noise-level underestimation [127]; the t-distribution is considered to handle the outliers [128]; and finally, instead of following the conditional mi”
Twenty Years of Mixture of Experts (2012, methodology): “5) Robot arm data, explained by Bishop [45].”
Mixtures of Experts Models (2018, methodology): “The reader is directed to Bishop & Svenskn (2003) for details on the variational approach; this section focuses on inference using MCMC methods.”
Regression Density Estimation With Variational Methods and Stochastic Approximation (2012, methodology): “Unlike the model with homoscedastic components (Bishop and Svensén 2003), not all the parameters have closed-form updates.”
Variational Message Passing (2005, methodology): “This can be taken a stage further (Bishop and Svensén, 2003) by using numerical differentiation applied to the lower bound.”
Bayesian reasoning and machine learning (2012, unknown): “See [290] for a variational treatment for regression and [43] for a variational treatment of classification.”
Deep Neural Decision Forests (2015, methodology): “In [2], a Bayesian approach using priors over all parameters is introduced, where also sigmoidal functions are used to model splits, based on linear functions on the input (c.”