Progressive Neural Architecture Search

By Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, Kevin P. Murphy. 2017

2,159 citations recorded by the source corpus.

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, methodology): “To accelerate NAS, one-shot models [2, 28, 31], DARTS [23] and NAS with Transferable Cell [21, 43] were proposed.”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Starting from 224x224 in early ConvNets, modern Con-vNets tend to use 299x299 (Szegedy et al., 2016) or 331x331 (Zoph et al., 2018) for better accuracy.”
Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “et RegNetY Figure 2: Evidence of overﬁtting on Imagenet-val: We compare the results obtained on ImageNet (x-axis) and the results obtained on ImageNet-v2 (y-axis), without FixRes for different models [16, 6, 14, 8, 2, 15, 4, 12]. For a given performance on Imagenet-val, overﬁtted models tend to have”
Fixing the train-test resolution discrepancy (2019, methodology): “We also conduct some experiments using the PNASNet-5-Large [24] architecture that exhibits good performance on ImageNet with a reasonable training time and number of parameters (86.”
Weight Standardization (2019, background): “Recently, Luo et al. [26] use the idea of AutoML [24, 48] to learn how to combine IN, LN, and BN to get a new normalization method.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “[53] Liu, C.”
Do ImageNet Classifiers Generalize to ImageNet? (2019, methodology): “pnasnet_large_tf [39] https://github.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “These systematic gains are similar to the improvements achieved with engineering new architectures [53, 22], however these gains arise without incurring additional computational cost at inference time.”
Designing Network Design Spaces (2020, methodology): “We observe that R EG N ETS are surprisingly effective in this regime considering the substantial body of work on ﬁnding better mobile networks via both manual design [9, 25, 19] and NAS [35, 23, 17, 18].”
On Network Design Spaces for Visual Recognition (2019, methodology): “As a case study of our methodology, we examine the network design spaces used by several recent methods for neural architecture search (NAS) [39, 29, 20, 28, 21].”
Squeeze-and-Excitation Networks (2017, methodology): “By formulating architecture search as hyperparameter optimisation, random search [34] and other more sophisticated model-based optimisation techniques [35], [36] can also be used to tackle the problem.”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “Other methods have also been applied: cascade-correlation [16], boosting [10], hill-climbing [14], MCTS [33], SMBO [28, 30], and random search [4], and grid search [49].”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “Concurrent to our work are the promising ideas of using performance prediction (Baker et al., 2017b; Deng et al., 2017), using iterative search method for architectures of growing complexity (Liu et al., 2017), and us-ing hierarchical representation of architectures (Liu et al., 2018).”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, methodology): “Recent advances in light-weight neural network architectures [2,8,18,21,24, 35,39] are mostly based on the metric of FLOPs and do not consider these properties above.”
DARTS: Differentiable Architecture Search (2018, methodology): “…approaches as it does not involve any controllers (Zoph and Le, 2016; Baker et al., 2016; Zoph et al., 2017; Pham et al., 2018b), hypernetworks (Brock et al., 2017) or performance predictors (Liu et al., 2017a), yet it is generic enough to search for both convolutional and recurrent architectures.”
The Evolved Transformer (2019, methodology): “Zela et. al’s (2018) utilization of Hyperband (Li et al., 2017) and PNAS’s (Liu et al., 2018a) incorporation of a surrogate model are examples of approaches that try to both increase efﬁciency via candidate performance estimation and maximize search quality by training models to the end when necessa”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, background): “These approaches are mainly based on reinforcement learning [35, 36, 1, 19, 25], evolutionary search [26], differentiable search [21], or other learning algorithms [19, 17, 23].”