Dynamic Slimmable Network

By Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, Xiaojun Chang. 2021

186 citations recorded by the source corpus.

Current dynamic networks and dynamic pruning methods have shown their promising capability in reducing theoretical computation complexity. However, dynamic sparse patterns on convolutional filters fail to achieve actual acceleration in real-world implementation, due to the extra burden of indexing, weight-copying, or zero-masking. Here, we explore a dynamic network slimming regime, named Dynamic Slimmable Network (DS-Net), which aims to achieve good hardware-efficiency via dynamically adjusting filter numbers of networks at test time with respect to different inputs, while keeping filters stored statically and contiguously in hardware to prevent the extra burden. Our DS-Net is empowered with the ability of dynamic inference by the proposed double-headed dynamic gate that comprises an attention head and a slimming head to predictively adjust network width with negligible extra computation cost. To ensure generality of each candidate architecture and the fairness of gate, we propose a disentangled two-stage training scheme inspired by one-shot NAS. In the first stage, a novel training technique for weight-sharing networks named In-place Ensemble Bootstrapping is proposed to improve the supernet training efficacy. In the second stage, Sandwich Gate Sparsification is proposed to assist the gate training by identifying easy and hard samples in an online way. Extensive experiments demonstrate our DS-Net consistently outperforms its static counterparts as well as state-of-the-art static and dynamic model compression methods by a large margin (up to 5.9%). Typically, DS-Net achieves 2-4× computation reduction and 1.62× real-world acceleration over ResNet-50 and MobileNet with minimal accuracy drops on ImageNet. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Xiaojun Chang
3. Yulin Wang
4. Gao Huang
5. Xiaodan Liang
6. Shirui Pan
7. Le Yang
8. Yizeng Han

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Xiaodan Liang
5. Xiaojun Chang
6. Yizeng Han
7. Le Yang
8. Guangrun Wang

In Model Architectures

1. Xiaodan Liang
2. Shiji Song
3. Yulin Wang
4. Gao Huang
5. Xiaojun Chang
6. Steven W. Su
7. Tao Tang
8. Honghui Wang

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Honghui Wang
7. Xiaojun Chang
8. Guangrun Wang

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “The recent work [89] uses a gate module to decide the width for a whole stage of a ResNet.”
• Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, unknown): “Previous works on gating mechanism [1, 25, 36, 44] employ binary gates, i.”
• BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, background): “Can-didates of different channel numbers in our supernet share the weights in a slimmable manner [78, 77, 76, 38, 9].”
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “…observe that DARTS is somewhat unreliable (Zela et al., 2020a; Chen & Hsieh, 2020; Li & Talwalkar, 2019; Sciuto et al., 2019; Zhang et al., 2020c; Li et al., 2021a; Zhang et al., 2020b) since it does not consistently yield excellent solutions, performing even worse than random search in some…”