Once for All: Train One Network and Specialize it for Efficient Deployment

By Han Cai, Chuang Gan, Song Han. 2019

1,586 citations recorded by the source corpus.

We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Furthermore, when training is decoupled from NAS, sub-networks tailoring to specialized system hardware can be selected without additional training [51], [52], [53], [54].”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, neutral): “As the proactive counterpart, Active Scene Description encompasses sources of feedback that are provided directly in response to active queries by the LLM planner, which are answered either by a person, or by another pretrained model, such as a Visual Question Answering (VQA) model [93, 94, 95, 96].”
On the Predictability of Pruning Across Scales (2020, background): “For example, Cai et al. [2019] create a single neural network architecture that can be scaled down to many different sizes; to choose which subnetwork to deploy, Cai et al. train an auxiliary, black-box neural network to predict subnetwork performance. Structure has also been observed—and, even furt”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “2% Neural Architecture Search (NAS) has brought significant improvements in many applications, such as machine perception [19, 48, 5, 51, 50], language modeling [32, 11], and model compression [17, 10, 16].”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “One method was pruning [9, 5, 46, 2], where either layers or channels were removed according to certain heuristics.”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “But more and more evidence shows that the parameter sharing strategy is likely to result in a sub-optimal inaccurate candidate architecture ranking [21, 24, 25, 99].”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “As a complementary direction, resource-aware NAS methods are proposed to simultaneously maximize prediction accuracy and minimize resource requirements such as latency, FLOPs, or memory footprints [4,9,30,31,33,34].”
Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “optimal cell architecture [9]–[13] or a whole network architecture [14]–[17].”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “Apart from evaluation, a recent work indicates that one-shot supernet weights can also specialize for submodels [3].”
ResNeSt: Split-Attention Networks (2020, background): “Recent work explored efficient neural architecture search via parameter sharing [40, 43] and have achieved great success in low-latency and low-complexity CNN models [3,57].”
ImageNet-21K Pretraining for the Masses (2021, methodology): “Unlike previous works, which focused on pretraining of large models only [24], we show that ImageNet-21K pretraining benefits a wide variety of models, from larger models like TResNet-L [38], through medium-sized models like ResNet50 [17], and even small mobile-dedicated models like OFA-595 [4] and ”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…at architecture-hardware co-design were manual, where an expert would adapt/change the NN architecture [70], followed by using automated AutoML and/or NAS techniques [22, 23, 99, 244]. c) Pruning : Another approach to reducing the memory footprint and computational cost of NNs is to apply pruning.”
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, methodology): “Some applied black-box optimization such as evolutionary search (Wang et al., 2020b) and reinforcement learning (Cai et al., 2019b; He et al., 2018; Wang et al., 2018, 2020a; Mao et al., 2019); Some leveraged backpropagation with differentiable architecture search (Liu et al., 2019).”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, background): “…the ﬁeld by removing human bias from architecture design (see, e.g., Elsken et al. (2019b)), achieve new state of the art on many tasks (Real et al., 2019; Chenxi et al., 2019; Saikia et al., 2019) and create resource-aware architectures (Tan et al., 2018; Elsken et al., 2019a; Cai et al., 2020).”
Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach (2021, positive): “In particular, inventors of the slimmable neural networks [46] proposed to train a single model to support layers with arbitrary widths; while authors of [18] proposed the “Once-for-All” architecture with a progressive shrinking algorithm which trains one network that supports diverse sub-networks.”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, background): “Model compression is a useful technology for deploying neural network models to mobile devices with limited storage and computational power [5, 6], and has attracted increasing attention.”