Universally Slimmable Networks and Improved Training Techniques

By Jiahui Yu, Thomas S. Huang. 2019

457 citations recorded by the source corpus.

Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: \url{https://github.com/JiahuiYu/slimmable_networks}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “To adapt the supernet for different expansion ratios, we train each operator with the largest width, the smallest width, and two random widths, following the “sandwich rule” in (Yu & Huang, 2019).”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “We also tried many other learning rate schedules with an exhaustive hyper-parameter sweep, including linearly decaying [24, 36] and cosine decaying [13, 23].”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, positive): “Recent hardware-aware NAS methods (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019) Huang, 2019b) propose to train a model to support multiple width multipliers (e.g., 4 different global width multipliers), building upon existing human-designed neural networks (e.g., MobileNetV2 0.35, 0.5, 0.75,”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, methodology): “There have been some works that train convolutional neural networks (CNNs) with adaptive width [32, 31, 30], and even both adaptive width and depth [2].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “The ﬁrst stage updates weights of unpruned networks with a variant sandwich rule [261].”
Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers (2019, methodology): “To train a slimmable model, we simply apply two width multipliers [7, 36] as the upper bound and lower bound of channel numbers.”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “[84] to construct switchable layers with shared weights but independent normalization layers, such that the corresponding layer can execute at different channel numbers from a predefined set, i.”
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, methodology): “In-place distillation [Yu and Huang, 2019] is applied to US-Nets.”
Any-Precision Deep Neural Networks (2019, background): “We observe active researches [20, 2, 4] in looking for more efﬁ-cient deep neural network architectures to support practical usage [15, 32, 26, 35].”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, background): “Candidates of different channel numbers in our supernet share the weights in a slimmable manner [77, 76, 75, 37].”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “The size search space is inspired by transformable architecture search methods [21], [31], [32].”
Towards Backward-Compatible Representation Learning (2020, background): “In [15], the authors discuss the possible mapping between feature vectors from multiple models trained on the same dataset; [45, 43, 44] introduce a design where multiple models with different channel widths but the same architecture share common subsets of parameters and representation, which impli”
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “Moreover, motivated by Inplace Knowledge Distillation (Yu & Huang, 2019b) in which a large model is used to supervise a small model, we propose to distill knowledge from the full-context mode (teacher) into the streaming mode (student) on the ﬂy within the same Universal ASR model, by encouraging…”
AutoSlim: Towards One-Shot Architecture Search for Channel Numbers (2019, methodology): “To train a slimmable model, we simply apply two width multipliers [7, 36] as the upper bound and lower bound of channel numbers.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “t with randomly assigned quantization precision for activation at every iteration. Note that the weights in the model are quantized to n W bits. et al. 2019a; Li et al. 2019; Phuong and Lampert 2019; Yu and Huang 2019; Zhang et al. 2019b). The difference between the proposed method and the previous ”
CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “The efficient block design [54, 53, 52], tensor decomposition [24, 49, 50, 42, 43], pruning [25, 26, 12], distillation [4, 48] and quantization [22] are popular techniques to make networks efficient.”
AOWS: Adaptive and Optimal Network Width Search With Latency Constraints (2020, methodology): “In section 3, we further detail these prior works [32, 33, 31], highlighting their similarities and differences with our approach, which we introduce in sections 4 to 6.”