Learning Transferable Architectures for Scalable Image Recognition
By Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. 2017
6,216 citations recorded by the source corpus.
Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (which we call the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, which we name a "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, a NASNet found by our method achieves 2.4% error rate, which is state-of-the-art. Although the cell is not searched for directly on ImageNet, a NASNet constructed from the best cell achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the image features learned from image classification are generically useful and can be transferred to other computer vision problems. On the task of object detection, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Dacheng Tao
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Yi Yang
- Alan L. Yuille
- Jitendra Malik
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Jiashi Feng
- Yoshua Bengio
- Shuicheng Yan
- Chunhua Shen
- Song Han
- Jianfeng Gao
Citations in the corpus
- A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained architectural principles at a larger scale (e.g., Liu et al., 2018; Real et al., 2018; Zoph et al., 2018).”
- Do Better ImageNet Models Transfer Better? (2018, methodology): “Specifically, we conduct a largescale study of transfer learning across 13 top-performing convolutional neural networks for image classification on 12 image classification datasets in 3 different experimental settings, visualized in Figure 1: as fixed feature extractors [2, 3], fine-tuned from Image”
- Exploring the Limits of Weakly Supervised Pretraining (2018, background): “5% boost over the prior state-of-the-art [31], which uses an optimized network architecture.”
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “beyond which no accuracy improvement can be obtained. Moreover, NAS is very computation and memory intensive. To accelerate NAS, one-shot models [2, 28, 31], DARTS [23] and NAS with Transferable Cell [21, 43] were proposed. The search time reduces dramatically but is still long from practical perspe”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Starting from 224x224 in early ConvNets, modern Con-vNets tend to use 299x299 (Szegedy et al., 2016) or 331x331 (Zoph et al., 2018) for better accuracy.”
- Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
- Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “Evidence of overfitting on Imagenet-val: We compare the results obtained on ImageNet (x-axis) and the results obtained on ImageNet-v2 (y-axis), without FixRes for different models [12, 13, 8, 14, 15, 2, 16, 1].”
- Self-Training With Noisy Student Improves ImageNet Classification (2019, neutral): “M - 79.0% 94.5% Inception-v4 [65] 48M - 80.0% 95.0% Inception-resnet-v2 [65] 56M - 80.1% 95.1% ResNeXt-101 [75] 84M - 80.9% 95.6% PolyNet [83] 92M - 81.3% 95.8% SENet [27] 146M - 82.7% 96.2% NASNet-A [86] 89M - 82.7% 96.2% AmoebaNet-A [54] 87M - 82.8% 96.1% PNASNet [39] 86M - 82.9% 96.2% AmoebaNet-C”
- Attention Augmented Convolutional Networks (2019, background): “Indeed, automated search strategies for designing architectures based on convolutional primitives result in state-of-the-art accuracy on large-scale image classification tasks that translate across a range of tasks [55, 21].”
- Relation Networks for Object Detection (2017, methodology): “The backbone network [42, 45, 43, 25, 8, 53] is pre-trained on ImageNet classification task [11] and fine-tuned during detection training.”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “[63] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “Various new architectures have been proposed since then, including VGG [24], NiN [16], Inception [1], ResNet [9], DenseNet [13], and NASNet [34].”
- VOLO: Vision Outlooker for Visual Recognition (2021, background): “EfficientNet [50] and NasNet [80] take advantage of neural architecture search to search for powerful network architectures.”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “These systematic gains are similar to the improvements achieved with engineering new architectures [53, 22], however these gains arise without incurring additional computational cost at inference time.”
- Fixing Weight Decay Regularization in Adam (2017, methodology): “SGDR has been successfully adopted to lead to new state-ofthe-art results for popular image classification benchmarks (Huang et al., 2017; Gastaldi, 2017; Zoph et al., 2017), and we therefore already tried extending it to Adam shortly after it was proposed.”
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, neutral): “Deep reinforcement learning methods have recently mastered a wide variety of domains through trial and error learning (Mnih et al., 2015; Silver et al., 2017; 2016; Zoph et al., 2017; Lillicrap et al., 2015; Barth-Maron et al., 2018).”
- Designing Network Design Spaces (2020, methodology): “We observe that R EG N ETS are surprisingly effective in this regime considering the substantial body of work on finding better mobile networks via both manual design [9, 25, 19] and NAS [35, 23, 17, 18].”
- On Network Design Spaces for Visual Recognition (2019, methodology): “The general NAS model family was introduced in [38, 39].”