HydraNets: Specialized Dynamic Architectures for Efficient Inference

By Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, Kayvon Fatahalian. 2018

160 citations recorded by the source corpus.

There is growing interest in improving the design of deep network architectures to be both accurate and low cost. This paper explores semantic specialization as a mechanism for improving the computational efficiency (accuracy-per-unit-cost) of inference in the context of image classification. Specifically, we propose a network architecture template called HydraNet, which enables state-of-the-art architectures for image classification to be transformed into dynamic architectures which exploit conditional execution for efficient inference. HydraNets are wide networks containing distinct components specialized to compute features for visually similar classes, but they retain efficiency by dynamically selecting only a small number of components to evaluate for any one input image. This design is made possible by a soft gating mechanism that encourages component specialization during training and accurately performs component selection during inference. We evaluate the HydraNet approach on both the CIFAR-100 and ImageNet classification tasks. On CIFAR, applying the HydraNet template to the ResNet and DenseNet family of models reduces inference cost by 2-4× while retaining the accuracy of the baseline architectures. On ImageNet, applying the HydraNet template improves accuracy up to 2.5% when compared to an efficient baseline architecture with similar inference cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Gao Huang
3. Shiji Song
4. Alan L. Yuille
5. Zhangyang Wang
6. Yoshua Bengio
7. Jianfeng Gao
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Alan L. Yuille
6. Chunhua Shen
7. Trevor Darrell
8. Jitendra Malik

In Model Architectures

1. Quoc V. Le
2. Zhangyang Wang
3. Samy Bengio
4. Yoshua Bengio
5. Gao Huang
6. Jifeng Dai
7. X. Zhang
8. Da-Cheng Juan

In Efficiency & Compression

1. X. Zhang
2. Shiji Song
3. Zhangyang Wang
4. Gao Huang
5. Asaf Noy
6. Tal Ridnik
7. Le Yang
8. Gabriel Bender

Citations in the corpus

• XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “Following works build upon this idea and include a gating mechanism per mixture [51, 41], and some further suggest sparsity regularization over experts via the gating mechanism [37, 44].”
• NBDT: Neural-Backed Decision Trees (2020, unknown): “In particular, several redesigned neural network architectures utilize decision tree branching structures [35,21,34].”
• Resolution Adaptive Networks for Efficient Inference (2020, background): “Furthermore, dynamically activating parts of network branches with multi-branch structure [35] also provide an alternate way for adaptive inference.”
• Improved Techniques for Training Adaptive Deep Networks (2019, methodology): “Existing works on adaptive inference in the context of deep learning mainly focus on designing more specialized network architectures [14, 33, 34] or better inference algorithms [2].”
• InstaNAS: Instance-aware Neural Architecture Search (2018, methodology): “Several conditional computation methods have been proposed to dynamically execute different modules of a model on a per-example basis [2, 12, 14, 27, 32, 30].”
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, background): “Mullapudi et al. [31] use clusters as labels to train a routing function between branches in a deep CNN model.”
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference (2020, background): “(Teja Mullapudi et al. 2018) selectively executes branches of network based on input.”
• Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, unknown): “[41] proposed”
• Learning Dynamic Routing for Semantic Segmentation (2020, methodology): “Traditional methods mainly focus on image classification by dropping blocks [38, 17, 33, 36] or pruning channels [39, 20] for efficient inference.”
• Channel Gating Neural Networks (2018, background): “[28] propose to replace the last residual block with a Mixture-of-Experts (MoE) layer and reduce computation cost by only activating a subset of experts in the MoE layer.”
• You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, background): “Particularly, several recent works [25, 28] propose to ensemble a small subset of dynamically selected experts in the model for each input.”