Regularized Evolution for Image Classifier Architecture Search

By Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V. Le. 2018

3,377 citations recorded by the source corpus.

The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted ones. Here, we evolve an image classifier— AmoebaNet-A—that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new state-of-theart 83.9% top-1 / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. This is relevant when fewer compute resources are available. Evolution is, thus, a simple method to effectively discover high-quality architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained architectural principles at a larger scale (e.g., Liu et al., 2018; Real et al., 2018; Zoph et al., 2018).”
Memory-Efficient Adaptive Optimization for Large-Scale Learning (2019, methodology): “Finally, we report results from a different domain: image classification on ImageNet [20] with the state-of-the-art AmoebaNet-D architecture [18], that has recently won the Stanford DAWNBench competition [9].”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “In fact, a few prior work (Zoph et al., 2018; Real et al., 2019) have already tried to arbitrarily balance network width and depth, but they all require tedious manual tuning.”
Towards a Human-like Open-Domain Chatbot (2020, methodology): “The Evolved Transformer is an evolutionary NAS architecture (Real et al., 2017, 2018) based on the Transformer (Vaswani et al., 2017).”
Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “[47] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V.”
True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “The search algorithm we use is Regularized Evolution [30] with hurdles [13].”
Designing Network Design Spaces (2020, background): “We observe that REGNETS are surprisingly effective in this regime considering the substantial body of work on finding better mobile networks via both manual design [7, 21, 16] and NAS [30, 19, 14, 15].”
On Network Design Spaces for Visual Recognition (2019, methodology): “Our main observation is that the EDFs vary considerably: the NASNet and Amoeba design spaces are noticeably worse than the others, while DARTS is best overall.”
Squeeze-and-Excitation Networks (2017, methodology): “While often computationally demanding, evolutionary search has had notable successes which include finding good memory cells for sequence models [27], [28] and learning sophisticated architectures for largescale image classification [29]–[31].”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “Our approach is inspired by recent advances in architecture search, where reinforcement learning and evolution have been used to discover model architectures from data [19, 21, 23, 33–41].”
Progressive Neural Architecture Search (2017, methodology): “Though the the EA method called “AmoebaNets” [19] currently give the highest accuracies (at the time of writing), it also requires the most compute, taking 63 times more resources than PNAS.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “The Evolved Transformer (So et al., 2019) was designed via evolution-based architecture search (Real et al., 2019) where the initial population was seeded with the original Transformer.”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, background): “• CIFAR-10 poses a diﬃcult enough problem so that the dataset is still the subject of active research (e.g., see [3, 4, 17, 21, 23]).”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, methodology): “Recent advances in light-weight neural network architectures [2,8,18,21,24, 35,39] are mostly based on the metric of FLOPs and do not consider these properties above.”
DARTS: Differentiable Architecture Search (2018, methodology): “This is a very interesting result, considering that so far the best architecture search methods used non-differentiable search techniques, e.g. based on RL (Zoph et al., 2017) or evolution (Real et al., 2018; Liu et al., 2017b We describe our search space in general form in Sect.”
The Evolved Transformer (2019, methodology): “The best performing architecture search methods are those that are computationally intensive (Zoph & Le, 2017; Baker et al., 2016; Real et al., 2017; Xie & Yuille, 2017; Zoph et al., 2018; Real et al., 2019).”