Convolution by Evolution: Differentiable Pattern Producing Networks

By Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, Daan Wierstra. 2016

106 citations recorded by the source corpus.

In this work we introduce a differentiable version of the Compositional Pattern Producing Network, called the DPPN. Unlike a standard CPPN, the topology of a DPPN is evolved but the weights are learned. A Lamarckian algorithm, that combines evolution and learning, produces DPPNs to reconstruct an image. Our main result is that DPPNs can be evolved/trained to compress the weights of a denoising autoencoder from 157684 to roughly 200 parameters, while achieving a reconstruction accuracy comparable to a fully connected network with more than two orders of magnitude more parameters. The regularization ability of the DPPN allows it to rediscover (approximate) convolutional network architectures embedded within a fully connected architecture. Such convolutional architectures are the current state of the art for many computer vision applications, so it is satisfying that DPPNs are capable of discovering this structure rather than having to build it in by design. DPPNs exhibit better generalization when tested on the Omniglot dataset after being trained on MNIST, than directly encoded fully connected autoencoders. DPPNs are therefore a new framework for integrating learning and evolution.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

HyperNetworks (2016, background): “Using this setting, hypernetworks can approximately re-2 cover the convolutional architecture without explicitly being told to do so, a similar result obtained by “Convolution by Evolution" (Fernando et al., 2016).”
Regularized Evolution for Image Classifier Architecture Search (2018, positive): “The traditional approach to architecture search is neuro-evolution of topologies (Angeline et al., 1994; Stanley & Miikkulainen, 2002; Floreano et al., 2008; Stanley et al., 2009; Fernando et al., 2016; Xie & Yuille, 2017).”
Large-Scale Evolution of Image Classifiers (2017, background): “…> . liest such “neuro-discovery” methods was neuro-evolution (Miller et al., 1989; Stanley & Miikkulainen, 2002; Stanley, 2007; Bayer et al., 2009; Stanley et al., 2009; Breuel & Shafait, 2010; Pugh & Stanley, 2013; Kim & Rigazio, 2015; Zaremba, 2015; Fernando et al., 2016; Morse & Stanley, 2016).”
AutoML: A Survey of the State-of-the-Art (2019, background): “some studies [116, 117, 118, 25] describe the use of indirect encoding schemes to represent the network.”
A genetic programming approach to designing convolutional neural network architectures (2017, background): “[6] proposed differentiable patternproducing networks (DPPNs) for optimizing the weights of a denoising autoencoder.”
Evolving Deep Neural Networks (2017, methodology): “Further, Fernando et al. [10] evolved a CPPN (compositional pattern-producing network [42]) to output the weights of an auto-encoder neural network. autoencoder was then trained further through gradient descent, forming gradients for the CPPN training, and its trained weights were then incorporated…”
Meta-learning autoencoders for few-shot prediction (2018, methodology): “[6] improves this method by making the weights differentiable, and Hypernetworks [9] further make both the network generator and network differentible.”
Population Based Training of Neural Networks (2017, methodology): “This mixture of learning and evolutionary-like algorithms has also been explored successfully in other domains (Zhang et al., 2011) such as neural architecture search (Real et al., 2017; Liu et al., 2017), feature selection (Xue et al., 2016), and parameter learning (Fernando et al., 2016).”
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search (2019, background): “Evolutionary Algorithm (EA) : Evolutionary algorithms represent each neural network as a string of genes and search the architecture space by mutation and recombinations [28, 38, 37, 13, 7, 20, 33, 43, 27, 39, 32].”
Evolution-Guided Policy Gradient in Reinforcement Learning (2018, background): “leveraged a similar idea to tackle catastrophic forgetting in transfer learning [17] and constructing differentiable pattern producing networks capable of discovering CNN architecture automatically [16].”
Continual Learning through Evolvable Neural Turing Machines (2016, background): “Given the increase in computational resources, this combination starts to be a promising research direction [16].”