A genetic programming approach to designing convolutional neural network architectures
By Masanori Suganuma, Shinichi Shirakawa, Tomoharu Nagao. 2017
618 citations recorded by the source corpus.
The convolutional neural network (CNN), which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, we attempt to automatically construct CNN architectures for an image classification task based on Cartesian genetic programming (CGP). In our method, we adopt highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The CNN structure and connectivity represented by the CGP encoding method are optimized to maximize the validation accuracy. To evaluate the proposed method, we constructed a CNN architecture for the image classification task with the CIFAR-10 dataset. The experimental result shows that the proposed method can be used to automatically find the competitive CNN architecture compared with state-of-the-art models.
Subfields: Artificial Intelligence, Model Architectures, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Model Architectures
In Neural Architecture Search & AutoML
- Frank Hutter
- Quoc V. Le
- Martin Wistuba
- Thomas Elsken
- Xia Hu
- Qingquan Song
- Jan Hendrik Metzen
- Yanan Sun
Citations in the corpus
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “Some methods [21]–[23] discard all models at regular intervals.”
- Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, background): “Noting that the proposed performance predictor is applicable to any existing EDLs [8]–[14].”
- Real-Time Federated Evolutionary Neural Architecture Search (2020, neutral): “Different from the previous neuro-evolution techniques [35], [36] that aim to optimize both the weights and architecture of neural networks, EA based NAS only optimizes the model architecture itself, and the model parameters are trained using conventional gradient descent methods [10]–[13].”
- A Survey on Neural Architecture Search (2019, methodology): “Suganuma et al. (2017) Suganuma et al. (2017) present another optimizer based on genetic algorithms but consider a wider set of operations in their search space, which includes convolutions and pooling along with concatenation and summation of vectors.”
- Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation (2019, background): “Early NAS works tried to search for a complete network topology [1, 29] while recent works focused on finding robust cells [33, 38, 23, 6].”
- A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification (2018, background): “Interested researchers have accomplished promising results on the automatic design of the architectures of CNNs by using Genetic Programming (GP) [17] and Genetic Algorithms (GAs) [21].”
- Auto-Keras: An Efficient Neural Architecture Search System (2018, background): “Many NAS approaches, such as deep reinforcement learning [2, 30, 40, 41], gradient-based methods [26] and evolutionary algorithms [10, 23, 31, 32, 34], require a large n to reach a good performance.”
- SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, background): “Evolutionary techniques [10, 37, 38, 40] offer a flexible approach for discovering variegated models from trivial initial conditions, but often struggle to scale to deep neural nets where the search space is vast, even with enormous compute power [26].”
- Practical Network Blocks Design with Q-Learning (2017, methodology): “In early research on automating neural network architecture design, most works relied on the genetic algorithm or other evolutionary algorithms (Schaffer, Whitley, and Eshelman 1992; Stanley and Miikkulainen 2002; Stanley, D’Ambrosio, and Gauci 2009; Suganuma, Shirakawa, and Nagao 2017).”
- Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “Whereas Real et al. (2017) also used enormous computational resources (250 GPUs, 10 days), Suganuma et al. (2017) were restricted to relatively small networks due to handling a population of networks.”
- AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search (2019, background): “Evolutionary Algorithm (EA) : Evolutionary algorithms represent each neural network as a string of genes and search the architecture space by mutation and recombinations [28, 38, 37, 13, 7, 20, 33, 43, 27, 39, 32].”
- Practical Neural Network Performance Prediction for Early Stopping (2017, background): “Suganuma et al. [20] use Cartesian genetic programming to obtain competitive results on image classification tasks.”
- Automatically Designing CNN Architectures Using Genetic Algorithm for Image Classification (2018, methodology): “Specifically, Genetic CNN [13], Hierarchical Evolution [15], EAS [18] and Block-QNN-S [21] belong to the second category, while Large-scale Evolution [14], CGP-CNN [16], NAS [18] and MetaQNN [19] fall into the third category.”
- Practical Block-Wise Neural Network Architecture Generation (2017, background): “Early works, from 1980s, have made efforts on automating neural network design which often searched good architecture by the genetic algorithm or other evolutionary algorithms [24, 27, 26, 28, 23, 7, 34].”
- A systematic review on overfitting control in shallow and deep neural networks (2021, methodology): “Suganuma et al. (2017) proposes a genetic algorithm to design a Convolutional Neural Network (CNN) architecture, where the fitness is the network accuracy.”
- Evolving convolutional neural networks through grammatical evolution (2019, methodology): “Similarly, for the CIFAR-10 (Table 2), GE-CNN achieved an accuracy of 76 , 34% on the best case and 72 , 84% on average, a very similar result compared to the CGP-CNN (ResSet) [9] approach, that achieved an accuracy of 76 , 53% on average.”