Large-Scale Evolution of Image Classifiers

By Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Y. Suematsu, Jie Tan, Quoc V. Le, Alexey Kurakin. 2017

1,766 citations recorded by the source corpus.

Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%, respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in the outcomes and the computational requirements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Neural Architecture Search (NAS) [39] and neural evolution [25, 1, 32, 21, 28] can search network architectures including layer depth, however, with very long search time.”
Towards a Human-like Open-Domain Chatbot (2020, methodology): “The Evolved Transformer is an evolutionary NAS architecture (Real et al., 2017, 2018) based on the Transformer (Vaswani et al., 2017).”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “Given a search space of TensorFlow programs, we use evolution [11, 12, 13, 14, 15, 16, 17] to search for models that achieve as low of a validation loss as possible given a ﬁxed amount of training compute.”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “A more flexible class of methods for designing architecture is evolutionary algorithms [65, 16, 57, 30, 46, 42, 67], yet they have not had as much success at large scale.”
Progressive Neural Architecture Search (2017, methodology): “Some of this work is summarized in Section 2, but at a high level, current techniques usually fall into one of two categories: evolutionary algorithms (see e.g. [28,24,35]) or reinforcement learning (see e.g., [40,41,39,5,2]).”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “Improved hardware now allows scaling up evolution to produce high-quality image classifiers (Real et al. 2017; Xie and Yuille 2017; Liu et al. 2018b).”
Efficient Neural Architecture Search via Parameter Sharing (2018, background): “ENAS’s design of sharing weights between architectures is inspired by the concept of weight inheritance in neural model evolution (Real et al., 2017; 2018).”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, methodology): “Recently, automatic model search [18,21,22,32,38,39] has become a promising trend for CNN architecture design.”
The Evolved Transformer (2019, methodology): “The best performing architecture search methods are those that are computationally intensive (Zoph & Le, 2017; Baker et al., 2016; Real et al., 2017; Xie & Yuille, 2017; Zoph et al., 2018; Real et al., 2019).”
Hierarchical Representations for Efficient Architecture Search (2017, methodology): “, 2017), Monte Carlo Tree Search (Negrinho & Gordon, 2017), evolution (Stanley & Miikkulainen, 2002; Xie & Yuille, 2017; Miikkulainen et al., 2017; Real et al., 2017), and reinforcement learning (Baker et al.”
Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “Thus, there has been an upsurge of research interest in neural architecture search (NAS), which automates the manual process of architecture design [1, 23].”
Weight Agnostic Neural Networks (2019, methodology): “In order to achieve state-of-the-art results, recent methods narrow the search space to architectures composed of basic building blocks with strong domain priors such as CNNs [64, 72, 89, 119], recurrent cells [48, 72, 119] and self-attention [100].”
Neural Architecture Optimization (2018, methodology): “Recently the design of neural network architectures has largely shifted from leveraging human knowledge to automatic methods, sometimes referred to as Neural Architecture Search (NAS) [45, 51, 52, 27, 37, 6, 40, 39, 28, 7, 6, 22].”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “Because of this, there is growing interest in automatic architecture search methods (Baker et al., 2017a; Zoph & Le, 2017; Real et al., 2017).”