Hierarchical Mixtures of Experts and the EM Algorithm

By Michael I. Jordan, Robert A. Jacobs. 1993

We present a tree-structured architecture for supervised learning. The statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models (GLIM's). Learning is treated as a maximum likelihood problem; in particular, we present an Expectation-Maximization (EM) algorithm for adjusting the parameters of the architecture. We also develop an on-line learning algorithm in which the parameters are updated incrementally. Comparative simulation results are presented in the robot dynamics domain.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “The recent surge in MoE language models builds on a relatively old idea (Jacobs et al., 1991; Jordan & Jacobs, 1994) of training ensembles of specialized models (“experts”) and a gating function to select the right expert for the task.”
A Review of Sparse Expert Models in Deep Learning (2022, background): “The concept of MoE in machine learning dates back at least three decades to the work of Jacobs et al. (1991); Jordan and Jacobs (1994).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “However, in the MoE we consider deterministic subset selection—determinism is a common assumption in MoE models (for example, in [14, 17, 33]) that can improve interpretability.”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Our work may be interpreted as a particular extension of Mixture-of-Experts in which experts may be composed to provide feedback to each other, closed-loop, via the common representation of language.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “…potential ties with conditional computation and mixture of experts methods, where specialized network sub-parts are active on a per-example basis (Jordan and Jacobs 1994; Eigen, Ranzato, and Sutskever 2014; Shazeer et al. 2017); we later provide evidence that FiLM learns to selectively highlight…”
Learning Factored Representations in a Deep Mixture of Experts (2013, background): “A product of experts (PoE) [6] is similar, but instead combines log probabilities to form a product: Also closely related to our work is the Hierarchical Mixture of Experts [9], which learns a hierarchy of gating networks in a tree structure.”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, background): “Mixture of experts (MoE) was ﬁrst proposed by Jacobs and Jordan [18].”
Distilling a Neural Network Into a Soft Decision Tree (2017, methodology): “This model is a hierarchical mixture of experts [Jordan and Jacobs, 1994], but each expert is a actually a bigot who does not look at the data after training, and therefore always produces the same distribution.”
Making Tree Ensembles Interpretable (2016, methodology): “M L ] 1 7 Ju n 20 16 perspective on ATMs, we can model an ATM as a mixtureof-experts (Jordan & Jacobs, 1994).”
Composing graphical models with neural networks for structured representations and fast inference (2016, methodology): “[1] [2] [3] [4] Gaussian mixture model Linear dynamical system Hidden Markov model Switching LDS [6] [2] [5] Mixture of Experts Driven LDS IO-HMM Factorial HMM [7]”
Learning Independent Causal Mechanisms (2017, background): “Early works on mixture of experts date back to the early nineties (Jacobs et al., 1991; Jordan & Jacobs, 1994), and since then the topic has been subject of extensive research.”
Combining Kernel and Model Based Learning for HIV Therapy Selection (2017, methodology): “The mixture-of-experts [9] is a technique that allows us to overcome the problems associated with choosing a particular”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, background): “Our work is also related to mixtures of experts architectures (Jacobs et al., 1991), (Jordan & Jacobs, 1994) as well as their modern attention based (Riemer et al., 2016) and sparse (Shazeer et al., 2017) variants.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “To address the issue that not all videos will have all descriptors, we design a model inspired by the mixture of experts [45], where we learn a separate “expert” embedding model for each descriptor type.”
Efficient Non-greedy Optimization of Decision Trees (2015, methodology): “The Hierarchical Mixture of Experts model [13] uses soft splits rather than hard binary decisions to capture situations where the transition from low to high response is gradual.”
Domain Adaptation for Statistical Classifiers (2006, background): “Most attempts to build discriminative mixture of experts models make heuristic approximations in order to perform the necessary optimization ( Jordan & Jacobs, 1994 ), rather than apply conditional EM, which gives us strict guarantees that we monotonically increase the data (incomplete) log likeliho”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “Routing networks are clearly related to task-decomposition modular networks [Jacobs et al., 1991a], and to mixtures of experts architectures [Jacobs et al., 1991b, Jordan and Jacobs, 1994] as well as their modern attention based [Riemer et al., 2016] and sparse [Shazeer et al., 2017] variants.”