Adaptive Mixtures of Local Experts

By Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, Geoffrey E. Hinton. 1991

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “Since its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research.”
Improving neural networks by preventing co-adaptation of feature detectors (2012, background): “For datasets in which the required input-output mapping has a number of fairly different regimes, performance can probably be further improved by making the dropout probabilities be a learned function of the input, thus creating a statistically efficient “mixture of experts” [14] in which there are ”
Distilling the Knowledge in a Neural Network (2015, background): “The use of specialists that are trained on subsets of the data has some resemblance to mixtures of experts [6] which use a gating network to compute the probabi lity of assigning each example to each expert.”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “(2017) name their architecture as MoE, it is not a standard MoE (Jacobs et al., 1991) and should be classified as MoC under our terminology.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “Many sparsely-activated networks have this property, but here we exclusively study the layout based on Sparse Mixtures of Experts [Shazeer et al., 2017] where multiple sub-components of a deep neural network (i.e., several layers) are independently converted to routed equivalents and jointly trained”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, methodology): “Several recent works modify transformer architecture by introducing sparse Mixture-of-Experts blocks (Jacobs et al., 1991; Shazeer et al., 2017).”
A Review of Sparse Expert Models in Deep Learning (2022, background): “The concept of MoE in machine learning dates back at least three decades to the work of Jacobs et al. (1991); Jordan and Jacobs (1994).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, positive): “The Mixture of Experts (MoE) [14] is the basis of many state-of-the-art deep learning models.”
Object-Centric Learning with Slot Attention (2020, background): “[63] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.”
Achieving Human Parity on Visual Question Answering (2021, methodology): “The methodology of Mixture of Experts (MoE) [31, 68] essentially decomposes a task into sub-tasks, and develops an expert on each sub-task.”
Dynamic Neural Networks: A Survey (2021, background): “MoE with soft weighting (a) and hard gating (b) schemes both adopt an auxiliary module to generate the weights or gates.”
Learning Factored Representations in a Deep Mixture of Experts (2013, background): “The Mixture of Experts model [7] is a continuous version of this: A learned gating network mixes the outputs of N “expert” networks to produce a ﬁnal output.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “Optimizing the ensemble weights, as in stacking [60] or adaptive mixture of experts [28], can further improve the performance.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, background): “Actually, the significance of diversity has been repeatedly demonstrated in previous work, such as inception modules [25], network ensembles [14], and a recent study [29] that relates ResNet with an implicit ensemble.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Besides, some regions of the biological brain have a modular architecture similar to the mixture of experts [183], such as the prefrontal cortex (i.”
Multiple-Source Adaptation with Domain Classifiers (2020, background): “Jacobs et al. [1991], Nowlan and Hinton [1991] considered an adaptive mixture of experts model, where there are multiple expert networks, as well as a gating network to determine which expert to use for each input.”
ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representations (2019, background): “xpert model to a given sample such that an improved performance is achieved. The expert matching problem is well-studied: (Jacobs et al. , 1991; Jacobs, 1995, 1997), starting from the seminal work of Jacobs et al. (1991), where the authors trained a versatile blend of experts for speaker vowel recog”
Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems (2020, background): “modules operate according to ﬁxed dynamics and each RIM module is specialized for a particular computation. RIM modules are thus not interchangeable. Neural Module Networks. A modular module network [Jacobs et al., 1991, Bottou and Gallinari, 1991, Ronco et al., 1997, Reed and De Freitas, 2015, Andr”
Recurrent Independent Mechanisms (2019, background): “…is composed of several neural modules, where each module is meant to perform a distinct function, and hence can be seen as a combination of experts (Jacobs et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2015; Andreas et al., 2016; Parascandolo et al., 2018;…”