A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks

By Kenneth O. Stanley, David B. D'Ambrosio, Jason Gauci. 2009

875 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Mechanical Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Mechanical Engineering

In Model Architectures

Citations in the corpus

HyperNetworks (2016, methodology): “HyperNEAT (Stanley et al., 2009) is an example of hypernetworks where the inputs are a set of virtual coordinates for each weight in the main network.”
Neural Architecture Search with Reinforcement Learning (2016, background): “In the last part of the section, we will focus on generating recurrent architectures, which is another key contribution of our paper.”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “roaches in designing architectures such as Neural Fabrics [48], DiffRNN [41], MetaQNN [3] and DeepArchitect [43]. A more ﬂexible class of methods for designing architecture is evolutionary algorithms [65, 16, 57, 30, 46, 42, 67], yet they have not had as much success at large scale. Xie and Yuille [”
Regularized Evolution for Image Classifier Architecture Search (2018, positive): “The traditional approach to architecture search is neuroevolution of topologies (Stanley & Miikkulainen, 2002; Floreano et al., 2008; Stanley et al., 2009).”
Large-Scale Evolution of Image Classifiers (2017, background): “Back-propagation and evolution can be combined as in Stanley et al. (2009), where only the structure is evolved.”
Hierarchical Representations for Efficient Architecture Search (2017, background): “The beneﬁts of indirect encoding schemes, such as multi-scale representations, have historically been discussed in Gruau et al. (1994); Kitano (1990); Stanley (2007); Stanley et al. (2009).”
Deep learning in neural networks: An overview (2014, neutral): “Instead of evolving large NNs directly (Section 6.6), one can sometimes greatly reduce the search space by evolving compact encodings of NNs, e.g., through Lindenmeyer Systems (Jacob, Lindenmayer, & Rozenberg, 1994; Lindenmayer, 1968), graph rewriting (Kitano, 1990), Cellular Encoding (Gruau, Whitle”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, neutral): “Many evolutionary approaches based on genetic algorithms are used to optimize the neural networks architecture and weights [117, 116, 2] while others rely on hierarchical evolution [74].”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “Early approaches [32, 31] use evolutionary algorithms to optimize both the neural architecture and its weights.”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “Another important approach to architecture search is neuroevolution (Floreano et al., 2008; Stanley et al., 2009; Real et al., 2017; Conti et al., 2017).”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, background): “Evolutionary techniques [10, 37, 38, 40] offer a ﬂexible approach for discovering variegated models from trivial initial conditions, but often struggle to scale to deep neural nets where the search space is vast, even with enormous compute power [26].”
A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “[35, 36] have optimized the weights of articial neural networks and CNN by using the hypercube-based neuroevolution of augmenting topologies (HyperNEAT) [31].”
Convolution by Evolution: Differentiable Pattern Producing Networks (2016, methodology): “In HyperNEAT [26], CPPNs are used as indirect compressed encodings of the weights of a larger neural network.”
Meta-learning autoencoders for few-shot prediction (2018, methodology): “The idea of using an indirect encoding for the weights of another network originates from the neu-roevolution algorithm of HyperNEAT [21].”
Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, methodology): “HyperNEAT (Stanley et al., 2009) is an indirect representation method that combines the NEAT algorithm with compositional patterns producing networks (CPPN) (Stanley, 2007) (III-A1, III-A2 and III-A3).”
Learning to Continually Learn (2020, methodology): “To avoid this explosion in the parameter count, one could harness indirect encoding, as in HyperNEAT [41], where a small network can create a geometric pattern (here, of gating) that can be applied to a larger network.”
Practical Network Blocks Design with Q-Learning (2017, methodology): “In early research on automating neural network architecture design, most works relied on the genetic algorithm or other evolutionary algorithms (Schaffer, Whitley, and Eshelman 1992; Stanley and Miikkulainen 2002; Stanley, D’Ambrosio, and Gauci 2009; Suganuma, Shirakawa, and Nagao 2017).”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, methodology): “Hyper-Neat Stanley et al. [2009] is an alternative approach to evolving both the weights of the neural networks and their parameters.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “However, the search space can often be reduced dramatically by evolving compact encodings of neural networks (NNs), e.g., through Lindenmeyer Systems [115], graph rewriting [127], Cellular Encoding [83], HyperNEAT [268], and other techniques [245, Sec.”