Mixture of experts: a literature survey

By Saeed Masoudnia, Reza Ebrahimpour. 2012

691 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “The notion as well of “Mixture-of-Experts” ((Jordan and Jacobs 1994), see (Masoudnia and Ebrahimpour 2014) for a review) is a common paradigm for combining the outputs of multiple models, and specifically mixtures of experts across multimodal domains including vision and audio (Liu et al.”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “In addition, parallels can be made with approaches utilizing mixture-of-experts (MoE) (Masoudnia & Ebrahimpour, 2014), where different examples are routed to different experts in order to maximize the output diversity (Shen et al., 2019b) or device utilization (Shazeer et al., 2017).”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “The decoder parameters are also shared, but its execution is taskspecific with each task token, with each forward akin to a task-specific expert chosen from a mixture of experts [52].”
Private Federated Learning with Domain Adaptation (2019, methodology): “Each user combines the output of the general and private models using a mixture of experts (MoE) (19; 21) to make their ﬁnal predictions.”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, positive): “Weadoptthemixtureofexperts[70]approach andusemultipleencodersasfeatureextractorstocalculate C .”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, background): “It builds 24 experts with 24 FC layers for data with different auxiliary label combinations ( 2 context-sensitivities, 3 data types, and 4 topics) in a mixture-of-expert manner (Masoudnia and Ebrahimpour, 2014).”
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models (2022, background): “MoE [17] is found to have strong ability in modern large-scale pre-trained models.”
AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style (2021, methodology): “…TTS system with fast and high-quality speech synthesis as well as powerful and efﬁcient adaptation capability; 2) an additional FP predictor, which is used to insert FP in the appropriate places in the sentences; 3) a newly designed duration predictor based on mixture of experts (MoE) [20, 21, 22].”
A Survey on ensemble learning under the era of deep learning (2021, methodology): “A popular variant of the Stacking method is the mixture of experts (ME) (Masoudnia and Ebrahimpour 2014), which first uses a divide-andconquer strategy to split the problem into multiple sub-problems, and respectively trains an expert (base learner) for each sub-problem, and then use another machine”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, methodology): “One promising way to reduce the training cost is using Mixture of Expert (MoE) [23].”
Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results (2020, methodology): “These weights are learned using a MoE architecture [37], so the entire model can be trained with gradient descent.”
Explaining Toxic Text via Knowledge Enhanced Text Generation (2022, methodology): “(Masoudnia and Ebrahimpour, 2012), which combine base expert model outputs into a final output using a gating mechanism.”
On the link between conscious function and general intelligence in humans and machines (2022, methodology): “While it is conspicuous to deal with diﬀerent sensory modalities via specializing processing, applying a divide-and-conquer paradigm is a common strategy more broadly within AI (Jacobs et al., 1991; Masoudnia & Ebrahimpour, 2014).”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, positive): “Eigen et al. (2013) introduced the idea of using multiple mixture-of-experts (Masoudnia & Ebrahimpour, 2014) with their own gating networks as parts of a deep model.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “This often takes the form of a mixture-of-experts [23, 24], in which a learned controller is used to select the experts relevant for a given input [25, 26, 27, 28, 29, 30], possibly hierarchically [31, 32, 33, 34].”
Mixtures of Experts Models (2018, background): “McLachlan & Peel (2000) and Frühwirth-Schnatter (2006) provide background to a range of mixtures of experts models; Masoudnia & Ebrahimpour (2014) survey the ME literature from a machine learning perspective.”
Advances in Intelligent Systems and Computing - Selected Papers from the International Conference on Computer Science and Information Technologies, CSIT 2016, September 6-10 Lviv, Ukraine (2017, methodology): “Numerous papers have been published to address deadlock handling strategies [1, 5-11].”
Interpretable and Differentially Private Predictions (2019, background): “MEs assign different specialized linear models to different parts of input space in a discriminative task (see Masoudnia and Ebrahimpour (2014) for an overview of existing ME models).”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “The MoE devotes to combining the local and diverse experts owning individual parameters for improving the overall accuracy and reliability [21], [22].”