Slimmable Neural Networks

By Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, Thomas S. Huang. 2018

657 citations recorded by the source corpus.

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, background): “Our work shares high-level similarities with slimmable networks [34–36] in terms of training a single shared set of weights which can be used for many child models.”
Real-Time Federated Evolutionary Neural Architecture Search (2020, background): “Apart from this, the trainable parameters in the batch normalization layer may slow down the convergence speed in federated learning, since they perform poorly for learning with small batch sizes [53] and weight sharing training paradigms [39], [54], [55], especially for non-IID scenarios.”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “, ResNet, MobileNetV2) [9, 28, 29], we have a much more diverse architecture space while allowing a significantly larger number of architectural configurations (10(19) v.”
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, methodology): “Beyond the NAS literature, our ﬁnding is closely related to Slimmable networks [23].”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “For example, in the scenarios where response time is more important than accuracy, some layers or channels could be abandoned at runtime for acceleration [39].”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some works that train convolutional neural networks (CNNs) with adaptive width [32, 31, 30], and even both adaptive width and depth [2].”
Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers (2019, methodology): “Our main idea lies in training a slimmable network [35] to approximate the network accuracy of different channel conﬁgurations.”
Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach (2021, background): “In particular, inventors of the slimmable neural networks [45] proposed to train a single model to support layers with arbitrary widths; while authors of [17] proposed the “Once-for-All” architecture with a progressive shrinking algorithm that trains one network to support diverse sub-networks.”
Dynamic Network Quantization for Efficient Video Inference (2021, positive): “To resolve mismatch in statistics of activations with different precisions, we use a separate set of Batch Normalization layers and clipping level parameters for different precisions [66].”
Dynamic Slimmable Network (2021, methodology): “Recently, slimmable networks [68, 66] using switchable batch normalization and in-place distillation achieve higher performance than their stand-alone”
Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, background): “The slimmable neural network [40] was recently proposed to train networks with varying channel widths while sharing parameters.”
WeightNet: Revisiting the Design Space of Weight Networks (2020, background): “Conditional CNNs Diﬀerent from standard CNNs [29,7,30,40,4,10,27,9], conditional (or dynamic) CNNs [17,20,37,39,15] use dynamic kernels, widths, or depths conditioned on the input samples, showing great improvement.”
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (2019, background): “Recent works [21, 27, 35, 19, 33, 49] focus on channel pruning in the CNNs, which removes entire weight filters instead of individual weights.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, background): “Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018; Figurnov et al., 2017; McGill & Perona, 2017; Teerapittayanon et al., 2016) Wang et al. (Wang et al., 2018) propose to skip…”
CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “The efficient block design [54, 53, 52], tensor decomposition [24, 49, 50, 42, 43], pruning [25, 26, 12], distillation [4, 48] and quantization [22] are popular techniques to make networks efficient.”
Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference (2020, neutral): “The slimmable neural network (Yu et al. 2018) trains the network with varied layer widths, and adjust channel number during inference to meet resource budgets.”
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions (2020, background): “Yu et al [38] amend this partially by creating a batchnorm invariant to the number input channels; after training the “supergraph” they see competitive accuracy without further training, for each possible subset of channels.”
Revisiting Dynamic Convolution via Matrix Decomposition (2021, background): “Yu et al. (2019) and Cai et al. (2019) train one network that supports multiple sub-networks of different complexities.”
Any-Precision Deep Neural Networks (2019, background): “[33] is related to ours in terms of methodology.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, positive): “Slimmable neural networks [72], [73] propose the switchable batch normalization, which permits models running with different channels.”