Generative NeuroEvolution for Deep Learning

By Phillip Verbancsics, Josh Harguess. 2013

54 citations recorded by the source corpus.

An important goal for the machine learning (ML) community is to create approaches that can learn solutions with human-level capability. One domain where humans have held a significant advantage is visual processing. A significant approach to addressing this gap has been machine learning approaches that are inspired from the natural systems, such as artificial neural networks (ANNs), evolutionary computation (EC), and generative and developmental systems (GDS). Research into deep learning has demonstrated that such architectures can achieve performance competitive with humans on some visual tasks; however, these systems have been primarily trained through supervised and unsupervised learning algorithms. Alternatively, research is showing that evolution may have a significant role in the development of visual systems. Thus this paper investigates the role neuro-evolution (NE) can take in deep learning. In particular, the Hypercube-based NeuroEvolution of Augmenting Topologies is a NE approach that can effectively learn large neural structures by training an indirect encoding that compresses the ANN weight pattern as a function of geometry. The results show that HyperNEAT struggles with performing image classification by itself, but can be effective in training a feature extractor that other ML approaches can learn from. Thus NeuroEvolution combined with other ML methods provides an intriguing area of research that can replicate the processes in nature.

Subfields: Artificial Intelligence, Biophysics, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computational Theory and Mathematics

Citations in the corpus

A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “[35, 36] have optimized the weights of artificial neural networks and CNN by using the hypercube-based neuroevo-”
Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “MetaQNN obtains an error of 6.92% as compared to 21.2% reported by Bergstra et al. (2011) on CIFAR-10; and it obtains an error of 0.32% as compared to 7.9% reported by Verbancsics & Harguess (2013) on MNIST.”
Convolution by Evolution: Differentiable Pattern Producing Networks (2016, methodology): “Recent work applied CPPNs in the HyperNEAT framework to evolve the weights of the 5 layer LeNet-5 convolutional neural network for MNIST character recognition [28].”
Evolving Deep Convolutional Neural Networks for Image Classification (2017, background): “Indeed, the views have been many years that evolutionary algorithms are incapable of evolving the architecture and connection weights in CNNs due to the tremendous number of related parameters [33]–[35].”
Practical Neural Network Performance Prediction for Early Stopping (2017, methodology): “The earliest meta-modeling approaches for neural net design were based on genetic algorithms [14, 19, 22].”
Large-Scale Evolution of Image Classifiers (2017, background): “Despite the promising results, the deep learning community generally perceives evolutionary algorithms to be incapable of matching the accuracies of hand-designed models (Verbancsics & Harguess, 2013; Baker et al., 2016; Zoph & Le, 2016).”
Accelerating Neural Architecture Search using Performance Prediction (2017, methodology): “Meta-modeling methods [2, 13, 1, 14] aim to design neural network architectures from scratch to reduce the amount of human expertise and labor usually required.”