FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function

By Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Péter Vajda and 1 more. 2020

77 citations recorded by the source corpus.

Neural Architecture Search (NAS) yields state-of-the-art neural networks that outperform their best manually-designed counterparts. However, previous NAS methods search for architectures under one training recipe (i.e., training hyperparameters), ignoring the significance of training recipes and overlooking superior architectures under other training recipes. Thus, they fail to find higher-accuracy architecture-recipe combinations. To address this oversight, we present JointNAS to search both (a) architectures and (b) their corresponding training recipes. To accomplish this, we introduce a neural acquisition function that scores architectures and training recipes jointly. Following pre-training on a proxy dataset, this acquisition function guides both coarse-grained and fine-grained searches to produce FBNetV3. FBNetV3 is a family of state-of-the-art compact ImageNet models, outperforming both automatically and manually-designed architectures. For example, FBNetV3 matches both EfficientNet and ResNeSt accuracy with 1.4x and 5.0x fewer FLOPs, respectively. Furthermore, the JointNAS-searched training recipe yields significant performance gains across different networks and tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Mingxing Tan
3. Ross B. Girshick
4. Xiaojun Chang
5. Trevor Darrell
6. Piotr Doll'ar
7. Xuanyi Dong
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Quoc V. Le
3. Luc Van Gool
4. Trevor Darrell
5. Dacheng Tao
6. Jitendra Malik
7. Xiaogang Wang
8. Alan L. Yuille

In Efficiency & Compression

1. Quoc V. Le
2. Mingxing Tan
3. Adams Wei Yu
4. Bogdan Gabrys
5. Mannat Singh
6. Ross B. Girshick
7. Piotr Doll'ar
8. Xuanyi Dong

In Neural Architecture Search & AutoML

1. Xiaojun Chang
2. Bogdan Gabrys
3. Quoc V. Le
4. Mingxing Tan
5. Daiyi Peng
6. Adams Wei Yu
7. Zhihui Li
8. Xin Wang

Citations in the corpus

• Early Convolutions Help Transformers See Better (2021, neutral): “0 S3 [3, 3, 3, 8, 1] [2, 1, 1, 8, 1] [1, 1, 1, 0, 0] [17, 34, 68, 136, 384] 458 0.”
• A Comprehensive Survey of Neural Architecture Search (2020, background): “Fortunately, the experiments of FBNetV3 [157] and AutoHAS [158] show that this direction has great potential, and it is expected to find higher-accuracy architecture-recipe combinations.”
• Fast and Accurate Model Scaling (2021, background): “2 [35], AutoAugment [4], stochastic weight averaging (SWA) [5], and mixed precision training [19].”
• AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Concurrent to our AutoHAS, FBNet-V3 (Dai et al., 2020) learns an acquisition function to predict the performance for the pair of hyperparameter and architecture.”