Twenty Years of Mixture of Experts

By Seniha Esen Yüksel, Joseph N. Wilson, Paul D. Gader. 2012

684 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Model Architectures

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “For a comprehensive review of the history of Mixture-of-Experts, predating the recent deep learning advances, we refer readers to the survey, “Twenty Years of Mixture-of-Experts” (Yuksel et al., 2012).”
Hash Layers For Large Sparse Models (2021, background): “To address these challenges, researchers have studied Mixtures-of-Experts (MoE) models [2, 3, 4, 5, 6, 7, 8], where a “gater” routes computation through a sparse subset of the weights of the model (the “expert modules”).”
Sparse-MLP: A Fully-MLP Architecture with Conditional Computation (2021, methodology): “Mixture-of-Experts(MoE) (Jacobs et al. 1991; Jordan and Jacobs 1994; Chen, Xu, and Chi 1999; Yuksel, Wilson, and Gader 2012; Shazeer et al. 2017) has been successfully applied to many domains and tasks (Gavrila and Munder 2006; Hu, Palreddy, and Tompkins 1997; Tani and Nolﬁ 1999; Sminchisescu et…”
A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Speciﬁcally, we identiﬁed two surveys preceding our work: the ﬁrst, published in August 2012 [31], provides a comprehensive review of early studies on dense MoE, which signiﬁcantly differ from the current mainstream focus on sparse MoE; the second, released in September 2022 [32], predates the…”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “In addition, to produce smooth predictions equipped with valid uncertainty, modeling averaging has been employed through mixture or product of experts [21]–[28].”
Inference for Subgroup Analysis With a Structured Logistic-Normal Mixture Model (2015, methodology): “Because such an EM algorithm has appeared in the literature for similar problems (Yuksel, Wilson, and Gader 2012), we forego the details.”
Generalizing DTW to the multi-dimensional case requires an adaptive approach (2016, neutral): “Our problem statement is superficially similar to a " gating network " in a Mixture of Experts (ME), a technique frequently used in neural networks and some other classifiers (Yuksel et al. 2012).”
Residual Mixture of Experts (2022, background): “Mixture of Experts (MoE) has a long history through these decades [6, 29, 31, 61].”
Scaling Vision with Sparse Mixture of Experts (2021, background): “MoEs [31, 34, 10, 66] combine the outputs of sub-models known as experts via a router in an input-dependent way.”
AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style (2021, methodology): “…TTS system with fast and high-quality speech synthesis as well as powerful and efﬁcient adaptation capability; 2) an additional FP predictor, which is used to insert FP in the appropriate places in the sentences; 3) a newly designed duration predictor based on mixture of experts (MoE) [20, 21, 22].”
On building ensembles of stacked denoising auto-encoding classifiers and their further improvement (2018, background): “Both of them could be applied to diversify DNNs, but their performance improvements are moderate for clas-siﬁcation, and their many modiﬁcations to get more advantage [47,48] demand higher computational effort, which, in principle, is not ideal when looking for methods to diversify DNNs.”
Multi-level Factorisation Net for Person Re-identification (2018, methodology): “Our MLFN architecture is also related to that of Mixtureof-Expert (MoE) models [13, 51, 1].”
Gated Multimodal Units for Information Fusion (2017, background): “However, the common usage of MoE is focused on performing decision fusion, i.e. combining predictors to address a supervised learning problem (Yuksel et al., 2012).”