Large scale evolution of convolutional neural networks using volunteer computing

By Travis Desell. 2017

77 citations recorded by the source corpus.

This work presents a new algorithm called evolutionary exploration of augmenting convolutional topologies (EXACT), which is capable of evolving the structure of convolutional neural networks (CNNs). EXACT is in part modeled after the neuroevolution of augmenting topologies (NEAT) algorithm, with notable exceptions to allow it to scale to large scale distributed computing environments and evolve networks with convolutional filters. In addition to multithreaded and MPI versions, EXACT has been implemented as part of a BOINC volunteer computing project, allowing large scale evolution. During a period of two months, over 4,500 volunteered computers on the Citizen Science Grid trained over 120,000 CNNs and evolved networks reaching 98.32% test data accuracy on the MNIST handwritten digits dataset. These results are even stronger as the backpropagation strategy used to train the CNNs was fairly rudimentary (ReLU units, L2 regularization and Nesterov momentum) and these were initial test runs done without refinement of the backpropagation hyperparameters. Further, the EXACT evolutionary strategy is independent of the method used to train the CNNs, so they could be further improved by advanced techniques like elastic distortions, pretraining and dropout. The evolved networks are also quite interesting, showing "organic" structures and significant differences from standard human designed architectures.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Auto-Keras: An Efficient Neural Architecture Search System (2018, background): “Many NAS approaches, such as deep reinforcement learning [2, 30, 40, 41], gradient-based methods [26] and evolutionary algorithms [10, 23, 31, 32, 34], require a large n to reach a good performance.”
Evolving Recurrent Neural Networks for Time Series Data Prediction of Coal Plant Parameters (2019, methodology): “2 that was shown by EXACT to improve training performance [3].”
Efficient Neural Architecture Search with Network Morphism (2018, background): “algorithms [23, 7, 18, 25, 29, 22], require a large n to reach a good performance.”
Auto-Keras: Efficient Neural Architecture Search with Network Morphism (2018, background): “, 2018) and evolutionary algorithms (Real et al., 2017; Desell, 2017; Liu et al., 2017; Suganuma et al., 2017; Xie & Yuille, 2017; Real et al., 2018), require a large n to reach a good performance.”
EDEN: Evolutionary deep networks for efficient machine learning (2017, methodology): “Desell [7] proposed EXACT, a neuro-evolutionary algorithm for deployment on a distributed cluster which they executed across 4500 volunteered computers and evolved 120,000 networks to tackle the MNIST dataset.”
Investigating recurrent neural network memory structures using neuro-evolution (2019, methodology): “The EXAMM algorithm presented in this work expands on an earlier algorithms, EXALT [11] which can evolve RNNs with either simple neurons or LSTM cells and EXACT which evolves convolutional neural networks for image classification [7, 8].”