Evolving Deep Neural Networks
By Risto Miikkulainen, J. Liang, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, B. Raju, H. Shahrzad, Arshak Navruzyan and 2 more. 2017
962 citations recorded by the source corpus.
Abstract The success of deep learning depends on finding an architecture to fit the task. As deep learning has scaled up to more challenging tasks, the architectures have become difficult to design by hand. This paper proposes an automated method, CoDeepNEAT, for optimizing deep learning architectures through evolution. By extending existing neuroevolution methods to topology, components, and hyperparameters, this method achieves results comparable to best human designs in standard benchmarks in object recognition and language modeling. It also supports building a real-world application of automated image captioning on a magazine website. Given the anticipated increases in available computing power, evolution of deep networks is promising approach to constructing deep learning applications in the future.
Subfields: Artificial Intelligence, Optimization & Training, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Sergey Levine
- Jianfeng Gao
- Trevor Darrell
- Alan L. Yuille
In Optimization & Training
- Quoc V. Le
- Frank Hutter
- Alan L. Yuille
- Barret Zoph
- Song Han
- Hanxiao Liu
- Martin Wistuba
- Jonathon Shlens
In Neural Architecture Search & AutoML
- Frank Hutter
- Martin Wistuba
- Quoc V. Le
- Song Han
- Thomas Elsken
- Tejaswini Pedapati
- Arber Zela
- Jan Hendrik Metzen
Citations in the corpus
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “lain4Net-6-6-6-6 in Table 11. DNNs in Table 11. Even with random initialization in new layers, the accuracy converges stably. 4 RELATED WORK Neural Architecture Search (NAS) [42] and neural evolution [1, 22, 26, 30, 34] can search network architectures from a gigantic search space. In NAS, the depth”
- Learning Transferable Architectures for Scalable Image Recognition (2017, background): “A more flexible class of methods for designing architecture is evolutionary algorithms [65, 16, 57, 30, 46, 42, 67], yet they have not had as much success at large scale.”
- Progressive Neural Architecture Search (2017, methodology): “RL is to use evolutionary algorithms (EA; \neuro-evolution" [32]). Early work (e.g., [33]) used EA to learn both the structure and the parameters of the network, but more recent methods, such as [28,24,35,21,27], just use EA to search the structures, and use SGD to estimate the parameters. RL a”
- Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “To speed up the process, researchers have looked into automated methods (Baker et al. 2017a; Zoph and Le 2016; Miikkulainen et al. 2017; Real et al. 2017; Xie and Yuille 2017; Suganuma, Shirakawa, and Nagao 2017; Liu et al. 2018a; Pham et al. 2018).”
- Hierarchical Representations for Efficient Architecture Search (2017, background): “…prediction (Brock et al., 2017), Monte Carlo Tree Search (Negrinho & Gordon, 2017), evolution (Stanley & Miikkulainen, 2002; Xie & Yuille, 2017; Miikkulainen et al., 2017; Real et al., 2017), and reinforcement learning (Baker et al., 2016; Zoph & Le, 2016; Zoph et al., 2017; Zhong et al.,…”
- AutoML: A Survey of the State-of-the-Art (2019, background): “There are two types of mutations in [22]: one enables or disables a connection between two layers, and the other adds or removes skip connections between two nodes or layers.”
- Weight Agnostic Neural Networks (2019, methodology): “In order to achieve state-of-the-art results, recent methods narrow the search space to architectures composed of basic building blocks with strong domain priors such as CNNs [64, 72, 89, 119], recurrent cells [48, 72, 119] and self-attention [100].”
- Neural Architecture Optimization (2018, methodology): “The latest algorithms for automatic architecture design usually fall into two categories: reinforcement learning (RL) [47, 48, 36, 3] based methods and evolutionary algorithm (EA) based methods [42, 34, 39, 27, 38].”
- Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture (Stanley & Miikkulainen, 2002; Liu et al., 2018a; Real et al., 2018; Miikkulainen et al., 2017; Xie & Yuille, 2017).”
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Orthogonally, a few studies benefit from extending the search space to two such aspects simultaneously (Zela et al., 2018; Miikkulainen et al., 2019; Noy et al., 2019).”
- Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, background): “However, NEAT has some limitations when it comes to evolving RNNs [32], e.g., the fitness landscape is deceptive and a large number of parameters have to be optimized.”
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “, 2018) or evolutionary algorithms (Stanley & Miikkulainen, 2002; Liu et al., 2018b; Miikkulainen et al., 2017; Real et al., 2017; 2019) to optimize the discrete architecture space.”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “The first type of architecture search methods (Liu et al., 2018b; Xie & Yuille, 2017; Real et al., 2017; Elsken et al., 2019; Real et al., 2019; Miikkulainen et al., 2019) adopted evolutionary algorithms, which assumed the possibility of applying genetic operations to force a single architecture or…”
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “LEAF uses CoDeepNEAT [85] which is a powerful evolutionary algorithm based on NEAT [104].”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “More recent neuroevolutionary approaches (Real et al., 2017; Liu et al., 2018b; Real et al., 2019; Miikkulainen et al., 2019; Xie & Yuille, 2017; Elsken et al., 2019a) use gradient-based methods for optimizing weights and solely use evolutionary algorithms for optimizing the neural architecture…”
- NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “The search algorithms mainly includes heuristic algorithm [19], [21], [32]–[34], reinforcement learning [35]–[38], [38], [39], the Bayesian optimization method [40], [41] and the gradient-based method [20], [42], [43].”
- Evaluating the Search Phase of Neural Architecture Search (2019, background): “Existing search strategies include reinforcement learning (RL) samplers [36, 37, 27], evolutionary algorithms [31, 28, 24, 17, 19], gradient-descent [18], and performance predictors [14, 20].”