Network of Experts for Large-Scale Image Categorization

By Karim Ahmed, Mohammad Haris Baig, Lorenzo Torresani. 2016

142 citations recorded by the source corpus.

We present a tree-structured network architecture for large-scale image classification. The trunk of the network contains convolutional layers optimized over all classes. At a given depth, the trunk splits into separate branches, each dedicated to discriminate a different subset of classes. Each branch acts as an expert classifying a set of categories that are difficult to tell apart, while the trunk provides common knowledge to all experts in the form of shared features. The training of our “network of experts” is completely end-to-end: the partition of categories into disjoint subsets is learned simultaneously with the parameters of the network trunk and the experts are trained jointly by minimizing a single learning objective over all classes. The proposed structure can be built from any existing convolutional neural network (CNN). We demonstrate its generality by adapting 4 popular CNNs for image categorization into the form of networks of experts. Our experiments on CIFAR100 and ImageNet show that in every case our method yields a substantial improvement in accuracy over the base CNN, and gives the best result achieved so far on CIFAR100. Finally, the improvement in accuracy comes at little additional cost: compared to the base network, the training time is only moderately increased and the number of parameters is comparable or in some cases even lower.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representations (2019, background): “, 2015; Ahmed et al. , 2016) by training one oracle model. Aljundi et al. (2017) learns a gating function to make expert network assignments. More recently, with the ongoing advances in machine learning and big data, Sharma et al. (2019) proposed a landscape for the expert matching problem and shows”
Expert Gate: Lifelong Learning with a Network of Experts (2016, background): “To avoid this issue, a mixture of one generalist model and many specialist models has been proposed [1, 13].”
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision (2017, neutral): “In [1, 19], the output of the model is the distribution given by taking the weighted some of the distributions of the experts by the distribution given by the generalist over the coarse classes.”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “We note that other techniques such as HD-CNN [46] and Network of Experts [1] also group related classes to perform hierarchical classiﬁcation, but these methods are not applicable for the multi-label setting (where labels are not mutually exclusive).”
Learning More Universal Representations for Transfer-Learning (2017, methodology): “thod, provides an answer to this limitation by carrying independent network training (one network per problem). From the point-of-view of the strategy used to vary the source problem, all the methods [1], [22], [35], [56] from this approach re-label speciﬁc categories into non-semantic generic ones,”
BranchConnect: Image Categorization with Learned Branch Connections (2017, methodology): “All these prior approaches [8, 24, 25, 1] differ from ours in the fact that they decompose the training into multiple stages, where first a full generalist is trained over all classes, and then separate expert networks or subnetworks are learned for different subsets of categories.”
Continual Learning in Neural Networks (2019, background): “To avoid this issue, a mixture of one generalist model and many specialist models has been proposed [3, 57].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “These methods all rely on some schemes such as branch-out [4] or selective learning [58].”
Deep Mixture of Experts via Shallow Embedding (2018, background): “More recently, several [22, 10, 1] have proposed MoE variants for deep learning in language modeling and image recognition domains.”
SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, background): “Recently, deep mixture of experts (MoE) based approaches [14, 15] have been intensively investigated and applied in different tasks such as language modeling [16, 17] and image classification [18, 19, 20, 21].”
SegNBDT: Visual Decision Rules for Segmentation (2020, background): “Interpretable Segmentation Models: Likwise, recent XAI work tackles directly interpretable classification models [32, 18, 23, 1, 4], but analogous work in segmentation is sparse.”
Multi-level Factorisation Net for Person Re-identification (2018, methodology): “Our MLFN architecture is also related to that of Mixtureof-Expert (MoE) models [13, 51, 1].”
NBDT: Neural-Backed Decision Trees (2020, methodology): “Our baselines are either taken directly from the original papers or improved using a modern backbone: Deep Neural Decision Forest (DNDF updated with ResNet18) [16], Explainable Observer-Classifier (XOC) [2], Deep Convolutional Decision Jungle (DCDJ) [3], Network of Experts (NofE) [1], Deep Decision ”
Scaling Vision with Sparse Mixture of Experts (2021, methodology): “For computer vision, previous work on MoEs [21, 2, 25, 1, 63, 47, 64] focused on architectures whose scale is considerably smaller than that of both language models and our model.”
HydraNets: Specialized Dynamic Architectures for Efficient Inference (2018, methodology): “Block 2 sep conv 3 × 3 [1, 2] 256 ×w sep conv 3 × 3 1 256 ×w”
Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles (2016, unknown): “[2] explores using ‘generalist’ network performance statistics to inform the design of ensemble-of-expert architectures for classification.”
ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries (2019, background): “In order to avoid the gating function, in (Hinton et al. , 2015; Ahmed et al. , 2016) trained a mixture of one oracle model that provides common knowledge to many specialist experts in the form of shared features.”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, background): “As a typical hierarchy, groupbased learning has been widely adopted in image classification tasks [18, 1, 49, 47, 12].”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “Then MoE has been applied to various machine learning algorithms (Jordan and Jacobs 1994; Collobert, Bengio, and Bengio 2003), such as SVMs (Collobert, Bengio, and Bengio 2002), Gaussian Processes (Tresp 2001), and deep networks (Ahmed, Baig, and Torresani 2016; Gu et al. 2018).”