Hard Mixtures of Experts for Large Scale Weakly Supervised Vision

By Sam Gross, Marc'Aurelio Ranzato, Arthur Szlam. 2017

128 citations recorded by the source corpus.

Training convolutional networks (CNNs) that fit on a single GPU with minibatch stochastic gradient descent has become effective in practice. However, there is still no effective method for training large networks that do not fit in the memory of a few GPU cards, or for parallelizing CNN training. In this work we show that a simple hard mixture of experts model can be efficiently trained to good effect on large scale hashtag (multilabel) prediction tasks. Mixture of experts models are not new [7, 3], but in the past, researchers have had to devise sophisticated methods to deal with data fragmentation. We show empirically that modern weakly supervised data sets are large enough to support naive partitioning schemes where each data point is assigned to a single expert. Because the experts are independent, training them in parallel is easy, and evaluation is cheap for the size of the model. Furthermore, we show that we can use a single decoding layer for all the experts, allowing a unified feature embedding space. We demonstrate that it is feasible (and in fact relatively painless) to train far larger models than could be practically trained with standard CNN architectures, and that the extra capacity can be well used on current datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

Citations in the corpus

Exploring the Limits of Weakly Supervised Pretraining (2018, background): “In addition to Flickr hashtags, [41] trained hard mixture of expert models on food-related Instagram hashtags; our focus is on standard recognition networks and general hashtags.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Various routing strategies (Gross et al., 2017; Lewis et al., 2021; Dua et al., 2021) have been investigated such that transfer learning can be achieved with task dependencies.”
Mixture-of-Experts with Expert Choice Routing (2022, background): “Hard MoE [12] with a single decoding layer can be efﬁciently trained to good effect on large scale hashtag prediction tasks.”
Large Memory Layers with Product Keys (2019, background): “Recently, there has been some efforts to develop high capacity architectures that operate on a limited computational budget [40, 18].”
Large scale distributed neural network training through online distillation (2018, background): “Gross et al. (2017) and Shazeer et al. (2017) are notable examples of recent work in this area.”
Hash Layers For Large Sparse Models (2021, background): “To address these challenges, researchers have studied Mixtures-of-Experts (MoE) models [2, 3, 4, 5, 6, 7, 8], where a “gater” routes computation through a sparse subset of the weights of the model (the “expert modules”).”
Scaling Vision with Sparse Mixture of Experts (2021, methodology): “For computer vision, previous work on MoEs [21, 2, 25, 1, 63, 47, 64] focused on architectures whose scale is considerably smaller than that of both language models and our model.”
ClusterFit: Improving Generalization of Visual Representations (2019, methodology): “…Supervised Learning: Training ConvNets on very large, weakly supervised images by deﬁning the proxy tasks using the associated meta-data [11, 20, 24, 31, 36, 38, 48, 51–53 , 56] has shown By contrast, our approach does not adhere to a particular target task and leverages all the data, since,…”
Speechmoe2: Mixture-of-Experts Model with Improved Routing (2021, background): “Recently, dynamic mixture-of-experts (MoE) based approaches have been investigated and applied in different tasks such as language modeling [1, 2] and image classiﬁcation [3, 4, 5, 6].”
CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, background): “Gross et al. [12] use a two stage training and clustering pipeline to train a hard mixture of experts model.”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, background): “As a typical hierarchy, groupbased learning has been widely adopted in image classification tasks [18, 1, 49, 47, 12].”
SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, background): “Recently, deep mixture of experts (MoE) based approaches [14, 15] have been intensively investigated and applied in different tasks such as language modeling [16, 17] and image classification [18, 19, 20, 21].”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Snowﬂake Arctic [29] employs a similar design, utilizing a Dense-MoE hybrid transformer architecture to effectively overlap communication with computation.”
Learning Layer-Skippable Inference Network (2020, unknown): “Different from the traditional MoEs that utilize sophisticated input-space partitioning method, [45] stated that clustering samples naively is enough for large-scale weakly supervised datasets.”
SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization (2020, methodology): “[29] used a gating network to save computation in very large networks; our scorer/gate mechanism also saves computation and memory by restricting attention to small subsets of the input data.”
Residual Mixture of Experts (2022, methodology): “MoE has been widely used in Natural Language Processing [18, 22, 35, 44] and computer vision [1, 16, 19, 41, 53, 58].”