Genetic CNN
By Lingxi Xie, Alan L. Yuille. 2017
931 citations recorded by the source corpus.
The deep convolutional neural network (CNN) is the state-of-the-art solution for large-scale visual recognition. Following some basic principles such as increasing network depth and constructing highway connections, researchers have manually designed a lot of fixed network architectures and verified their effectiveness.,,In this paper, we discuss the possibility of learning deep network structures automatically. Note that the number of possible network structures increases exponentially with the number of layers in the network, which motivates us to adopt the genetic algorithm to efficiently explore this large search space. The core idea is to propose an encoding method to represent each network structure in a fixed-length binary string. The genetic algorithm is initialized by generating a set of randomized individuals. In each generation, we define standard genetic operations, e.g., selection, mutation and crossover, to generate competitive individuals and eliminate weak ones. The competitiveness of each individual is defined as its recognition accuracy, which is obtained via a standalone training process on a reference dataset. We run the genetic process on CIFAR10, a small-scale dataset, demonstrating its ability to find high-quality structures which are little studied before. The learned powerful structures are also transferrable to the ILSVRC2012 dataset for large-scale visual recognition.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Plant Science, Neural Architecture Search & AutoML, Evolutionary Computation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Dacheng Tao
- Alan L. Yuille
- Jianfeng Gao
- Trevor Darrell
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Trevor Darrell
- Alan L. Yuille
- Chunhua Shen
In Plant Science
In Neural Architecture Search & AutoML
- Frank Hutter
- Martin Wistuba
- Colin White
- Yash Savani
- Xiangxiang Chu
- Xia Hu
- Qingquan Song
- Willie Neiswanger
In Evolutionary Computation
Citations in the corpus
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “So in [23], instead of using each bit as a unit, the basic unit in the crossover is a stage, which is a higher-level structure constructed by a binary string.”
- Neural Architecture Optimization (2018, methodology): “The latest algorithms for automatic architecture design usually fall into two categories: reinforcement learning (RL) [47, 48, 36, 3] based methods and evolutionary algorithm (EA) based methods [42, 34, 39, 27, 38].”
- Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture (Stanley & Miikkulainen, 2002; Liu et al., 2018a; Real et al., 2018; Miikkulainen et al., 2017; Xie & Yuille, 2017).”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “2% Neural Architecture Search (NAS) has brought significant improvements in many applications, such as machine perception [19, 48, 5, 51, 50], language modeling [32, 11], and model compression [17, 10, 16].”
- A Comprehensive Survey of Neural Architecture Search (2020, background): “architecture by comparing the performance of a large number of candidate network architectures. Therefore, the performance ranking of candidate network architectures is extremely important. Early NAS [11, 12, 15, 16] usually fully trained the candidate network architecture, and then obtained the ran”
- Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, background): “consume less computational resource than other CNN architecture design algorithms [11]–[14], while still can achieve the promising results.”
- Towards modular and programmable architecture search (2019, methodology): “We use the Genetic search space [26] for these experiments as Figure 8 shows its architectures train quickly and have substantially different validation accuracies.”
- Auto-GNN: Neural architecture search of graph neural networks (2019, background): “Most of NAS frameworks are built up based on one of the two basic algorithms: RL [12, 13, 15, 38, 39] and EA [14, 40–43].”
- AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, methodology): “For the optimization algorithm, popular options include reinforcement learning [4, 71, 69, 72], evolutionary algorithm [60], Bayesian optimization [24], random search [7], and gradient-based optimization methods [38, 1].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “Among them, Liu et al. (Liu et al., 2018b) introduced a hierarchical representation for describing a network architecture, and Xie et al. (Xie & Yuille, 2017) decomposed each architecture into a representation of ‘genes’.”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “s.NE] 27 Jun 2019 This would confer not only generality but also easy of use. Most of the algorithms for AAS are either based on reinforcement learning [44],[45],[27], [3] or evolutionary computation [41], [22], [29],[17], [28]. On the one hand, in reinforcement learning approaches, architectures ar”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “More recent neuroevolutionary approaches (Real et al., 2017; Liu et al., 2018b; Real et al., 2019; Miikkulainen et al., 2019; Xie & Yuille, 2017; Elsken et al., 2019a) use gradient-based methods for optimizing weights and solely use evolutionary algorithms for optimizing the neural architecture…”
- A Survey on Neural Architecture Search (2019, methodology): “Xie and Yuille (2017) Xie and Yuille (2017) made their work available only one day later.”
- Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation (2019, background): “Early NAS works tried to search for a complete network topology [1, 29] while recent works focused on finding robust cells [33, 38, 23, 6].”
- NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “The search algorithms mainly includes heuristic algorithm [19], [21], [32]–[34], reinforcement learning [35]–[38], [38], [39], the Bayesian optimization method [40], [41] and the gradient-based method [20], [42], [43].”
- Evaluating the Search Phase of Neural Architecture Search (2019, background): “Existing search strategies include reinforcement learning (RL) samplers [36, 37, 27], evolutionary algorithms [31, 28, 24, 17, 19], gradient-descent [18], and performance predictors [14, 20].”
- Overcoming Multi-Model Forgetting (2019, neutral): “Deep neural networks have been very successful for tasks such as visual recognition (Xie & Yuille, 2017) and nat-ural language processing (Young et al., 2017), and much recent work has addressed the training of models that can generalize across multiple tasks (Caruana, 1997).”
- Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “An alternative to RL is to use evolutionary algorithms (EA), that “evolves” architectures by mutating the best architectures found so far [62, 83, 53, 48, 61].”
- Graph HyperNetworks for Neural Architecture Search (2018, methodology): “…learning (Zoph & Le, 2017; Baker et al., 2017a; Zoph et al., 2018), evolutionary methods (Real et al., 2017; Miikkulainen et al., 2017; Xie & Yuille, 2017; Liu et al., 2018b; Real et al., 2018) and gradient-based methods (Liu et al., 2018c; Luo et al., 2018) have been proposed to…”