Dynamic Routing Networks

By Shaofeng Cai, Yao Shu. 2019

60 citations recorded by the source corpus.

The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expensive models that only a small portion of input instances really need. Inference with a static architecture that processes all input instances via the same transformation would thus incur unnecessary computational costs. Therefore, customizing the model capacity in an instance-aware manner is much needed for higher inference efficiency. In this paper, we propose Dynamic Routing Networks (DRNets), which support efficient instance-aware inference by routing the input instance to only necessary transformation branches selected from a candidate set of branches for each connection between transformation nodes. The branch selection is dynamically determined via the corresponding branch importance weights, which are first generated from lightweight hyper-networks (RouterNets) and then recalibrated with Gumbel-Softmax before the selection. Extensive experiments show that DRNets can reduce a substantial amount of parameter size and FLOPs during inference with prediction performance comparable to state-of-the-art architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Dong Yu
5. Dan Su
6. Le Yang
7. Yizeng Han
8. Zhao You

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Yizeng Han
5. Le Yang
6. Dong Yu
7. Honghui Wang
8. Dan Su

In Model Architectures

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Dong Yu
5. Dan Su
6. Zhao You
7. Honghui Wang
8. Le Yang

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “For another example, dynamic routing network (DRNet) [80] performs a branch selection in each cell structure which is commonly used in NAS [10].”
• Speechmoe2: Mixture-of-Experts Model with Improved Routing (2021, background): “Recently, dynamic mixture-of-experts (MoE) based approaches have been investigated and applied in different tasks such as language modeling [1, 2] and image classification [3, 4, 5, 6].”
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, background): “Recently, deep mixture of experts (MoE) based approaches [14, 15] have been intensively investigated and applied in different tasks such as language modeling [16, 17] and image classification [18, 19, 20, 21].”