BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models

By Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas S. Huang, Xiaodan Song, Quoc V. Le. 2020

335 citations recorded by the source corpus.

Neural architecture search (NAS) has shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has become a popular strategy to rank the relative quality of different architectures (child models) using a single set of shared weights. However, while one-shot model weights can effectively rank different network architectures, the absolute accuracies from these shared weights are typically far below those obtained from stand-alone training. To compensate, existing methods assume that the weights must be retrained, finetuned, or otherwise post-processed after the search is completed. These steps significantly increase the compute requirements and complexity of the architecture search and model deployment. In this work, we propose BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies. Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing state-of-the-art models in this range including EfficientNets and Once-for-All networks without extra retraining or post-processing. We present ablative study and analysis to further understand the proposed BigNASModels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Neural Architecture Search & AutoML

Citations in the corpus

AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, background): “Recently, to improve the ﬁnal performance of the searched architectures, Yu et al. (2020) utilizes knowledge distillation which is orthogonal to our method.”
Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, background): “, 2018a), and weight-sharing (Cai et al., 2020; Yu et al., 2020; Wang et al., 2021), among others.”
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, background): “Recent works further acquired scalability in depth [Zhang et al., 2019], resolution [Cai et al., 2020; Yu et al., 2020; Yang et al., 2020], and kernel size [Cai et al., 2020; Yu et al., 2020].”
ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding (2022, neutral): “It is also expected to benefit from modern training and boosting techniques like network architecture search [34,53] and loss function search [30,46], though they are out of the bound of this work.”
Sample-Efficient Neural Architecture Search by Learning Action Space (2019, neutral): “8 BigNAS ([43]) 242M 4.”
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, background): “Another related research work in Computer Vision is Slimmable Neural Networks (Yu et al., 2018; Yu & Huang, 2019a;b; Yu et al., 2020).”
NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization (2021, methodology): “3), and superkernel [12, 21, 22] to efficiently train the super-network that contains sub-networks with different layer widths, network depths, and kernel sizes.”
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, methodology): “BigNAS [7], proposes inplace distillation with a sandwich rule to supervise the training of subnetworks by the largest child model.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “However, most works suffer from heavy search cost, for both weight-sharing based methods (Liu et al., 2018b; Dong & Yang, 2019; Liu et al., 2019; Yu et al., 2020a; Li et al., 2020a; Yang et al., 2020a) and single-path sampling-based methods (Pham et al., 2018; Guo et al., 2019; Real et al., 2019;…”
AutoFormer: Searching Transformers for Visual Recognition (2021, methodology): “Moreover, BigNAS adopts several well-crafted techniques, such as sandwich training, inplace distillation, regularization, etc .”
Parameter Prediction for Unseen Deep Architectures (2021, background): “…12, 103] that can be used to estimate the performance of smaller nets (subnets) obtained by some kind of pruning the supernet, followed by training the best chosen a from scratch with SGD. Recent models, in particular BigNAS [12] and OnceForAll (OFA) [102], eliminate the need to train subnets.”
Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning (2020, methodology): “Therefore, we employ the weight sharing technique [4, 31, 50] Network Info possible sub-nets (i.e., different combinations of weight operations).”
Dynamic Slimmable Network (2021, positive): “Some recent works [6, 73, 24] also explore anytime neural networks in multiple dimensions, e.g. depth, width, kernel size, etc. Dynamic neural networks [63, 64, 46, 68] change their architectures based on the input data.”