Learning Factored Representations in a Deep Mixture of Experts

By David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever. 2013

490 citations recorded by the source corpus.

Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associating each input with a combination of experts at each layer, yet maintains a modest model size. On a randomly translated version of the MNIST dataset, we find that the Deep Mixture of Experts automatically learns to develop location-dependent ("where") experts at the first layer, and class-specific ("what") experts at the second layer. In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones. These demonstrate effective use of all expert combinations.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model.”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “6 The target of Eigen et al. (2013) is to create a more expressive feed-forward layer through the mixture structure.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “Many sparsely-activated networks have this property, but here we exclusively study the layout based on Sparse Mixtures of Experts [Shazeer et al., 2017] where multiple sub-components of a deep neural network (i.e., several layers) are independently converted to routed equivalents and jointly trained”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “Gating techniques relying on sampling and straight-through gradient estimators are common (Bengio et al., 2013; Eigen et al., 2013; Cho and Bengio, 2014).”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “The output computation of the layer is the linearly weighted combination of each expert’s computation on the token by the gate value, We note that in contrast to Eigen et al. (2013), this selection is only over the top-k experts and is thus more computationally efﬁcient.”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “To facilitate such routing, different approaches have been developed, including large mixture of experts (Shazeer et al., 2017) or gating (Eigen et al., 2013; Cho and Bengio, 2014).”
Large Memory Layers with Product Keys (2019, methodology): “Different methods have been developed like large mixture of experts [39], gating techniques [4, 12, 6] or even reinforcement learning-based approaches [10].”
Dynamic Neural Networks: A Survey (2021, background): “Conventional soft MoEs [41], [76], [77] adopt real-valued weights to dynamically rescale the representations obtained”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, background): “…computation and mixture of experts methods, where specialized network sub-parts are active on a per-example basis (Jordan and Jacobs 1994; Eigen, Ranzato, and Sutskever 2014; Shazeer et al. 2017); we later provide evidence that FiLM learns to selectively highlight or suppress feature…”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Gradient Episodic Memory for Continual Learning (2017, methodology): “This relates to methods that leverage a modular structure of the network with primitives that can be shared across tasks [Rusu et al., 2016, Fernando et al., 2017, Aljundi et al., 2016, Denoyer and Gallinari, 2015, Eigen et al., 2014].”
Efficient Continual Learning with Modular Networks and Task-Driven Priors (2020, neutral): “It is based on a modular neural network architecture (Eigen et al., 2014; Denoyer & Gallinari, 2015; Fernando et al., 2017; Li et al., 2019) with a novel task-driven prior (§4).”
Opponent Modeling in Deep Reinforcement Learning (2016, background): “For example, Eigen et al. (2014) extends the Mixture-of-Experts network to a stacked model—deep Mixture-of-Experts—which can be combined with hierarchical reinforcement learning to learn a hierarchy of opponent strategies in large, complex domains such as online strategy games.”
Hash Layers For Large Sparse Models (2021, background): “To address these challenges, researchers have studied Mixtures-of-Experts (MoE) models [2, 3, 4, 5, 6, 7, 8], where a “gater” routes computation through a sparse subset of the weights of the model (the “expert modules”).”
Modular Networks: Learning to Decompose Neural Computation (2018, background): “The latter idea has been, in part, pursued by [9], relying on stacked mixtures of experts instead of a single selection mechanism.”
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts (2018, methodology): “MoELayer :WhileMoEwas rst developed as an ensemblemethod of multiple individual models, Eigen et al [16] and Shazeer et al [31] turn it into basic building blocks (MoE layer) and stack them in a DNN.”
XNAS: Neural Architecture Search with Expert Advice (2019, background): “A different concept of using multiple mixtures of experts as inner parts of a deep model, where each mixture has its own gating network, is presented in [10].”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, background): “…naı̈ve implementations of mixture models for text generation are prone to two major types of degeneracies: D1: Only one component gets trained (Eigen et al., 2014; Shazeer et al., 2017) because of the “rich gets richer” effect whereby, once a component is slightly better than others, it is…”