Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers

By Jiahui Yu, Thomas S. Huang. 2019

59 citations recorded by the source corpus.

We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot solution, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs). Code and models will be available at: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “Our work shares high-level similarities with slimmable networks [36–38] in terms of training a single shared set of weights which can be used for many child models.”
Network Pruning via Transformable Architecture Search (2019, background): “A possible solution to accelerate ES or random searching methods is to share parameters of networks with different configurations [39, 45], which is beyond the scope of this paper.”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some works that train convolutional neural networks (CNNs) with adaptive width [32, 31, 30], and even both adaptive width and depth [2].”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “To alleviate this problem, we apply batch norm (BN) calibration introduced in [47] to calibrate the statistics in batch normalization layers for reasonable comparison.”
Towards Backward-Compatible Representation Learning (2020, background): “In [15], the authors discuss the possible mapping between feature vectors from multiple models trained on the same dataset; [45, 43, 44] introduce a design where multiple models with different channel widths but the same architecture share common subsets of parameters and representation, which impli”
Neural Architecture Search with GBDT (2020, background): “Neural architecture search (NAS) has shown its effectiveness in automatically designing neural network architectures and been applied in many tasks such as image classiﬁcation [37, 38], object detection [6, 4], network pruning [33] and neural machine translation [22].”
Universally Slimmable Networks and Improved Training Techniques (2019, background): “Our discovery opens up the possibility to many related fields, for examples, network comparison in terms of FLOPs-Accuracy spectrum (Figure 1), and one-shot architecture search for number of channels [24].”