Evolving Neural Networks through Augmenting Topologies

By Kenneth O. Stanley, Risto Miikkulainen. 2002

3,694 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Neural Architecture Search & AutoML

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “lain4Net-6-6-6-6 in Table 11. DNNs in Table 11. Even with random initialization in new layers, the accuracy converges stably. 4 RELATED WORK Neural Architecture Search (NAS) [42] and neural evolution [1, 22, 26, 30, 34] can search network architectures from a gigantic search space. In NAS, the depth”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, neutral): “In contrast to early work on adaptively growing networks, e.g., growing neural gas (Fritzke, 1995) and neuro evolution of augmenting topologies (Stanley & Miikkulainen, 2002) that grow networks greedily, we simply defer the introduction of pre-conﬁgured layers.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “We ﬁnd that the inverted pendulum task is surprisingly easy, with d int100 = d int90 = 4 , meaning that only four parameters are needed to perfectly solve the problem (see Stanley & Miikkulainen (2002) for a similarly small solution found via evolution).”
Squeeze-and-Excitation Networks (2017, methodology): “Much of the early work in this domain was conducted in the neuro-evolution community, which established methods for searching across network topologies with evolutionary methods [25], [26].”
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (2014, background): “3, CPPN networks start with no hidden nodes, and nodes are added over time, encouraging evolution to first search for simple, regular images before adding complexity [27].”
Progressive Neural Architecture Search (2017, methodology): “Other related works include [23], who focus on MLP rather than CNNs; [33], who used an incremental approach in the context of evolutionary algorithms; [40] who used a schedule of increasing number of layers; and [13] who search through the space of latent factor models speciﬁed by a grammar.”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, background): “Residual Networks (ResNet) [13] and Highway Networks [29] each propose the use of connections that skip over multiple layers, for example additively connecting the activations from layer 3 to the activations from layer 6.”
Large-Scale Evolution of Image Classifiers (2017, methodology): “Correspondence to: Esteban Real < ereal@google.com > . liest such “neuro-discovery” methods was neuro-evolution (Miller et al., 1989; Stanley & Miikkulainen, 2002; Stanley, 2007; Bayer et al., 2009; Stanley et al., 2009; Breuel & Shafait, 2010; Pugh & Stanley, 2013; Kim & Rigazio, 2015; Zaremba,…”
Hierarchical Representations for Efficient Architecture Search (2017, methodology): “In contrast to several previous works where genotypes are initialized by trivial networks (Stanley & Miikkulainen, 2002; Real et al., 2017) the above diversiﬁcation-based scheme not only offers a good initial coverage of the search space with non-trivial architectures, but also helps to avoid an…”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “NEAT performs artiﬁcial synapsis based on historical markings, allowing it to add new structure without losing track of which gene is which throughout a simulation.”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “Reinforcement-learning-based NAS is end-to-end for gradient back-propagation, among which the most efﬁcient one, ENAS (Pham et al., 2018) learns optimal parameters and architectures together just like NEAT.”
Weight Agnostic Neural Networks (2019, methodology): “Our NumPy [111] implementation of NEAT [103] supports MPI [32] and OpenAI Gym [10] environments.”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture (Stanley & Miikkulainen, 2002; Liu et al., 2018a; Real et al., 2018; Miikkulainen et al., 2017; Xie & Yuille, 2017).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “AutoML has utilized a wide array of paradigms, including growing networks neuron-by-neuron (Stanley & Miikkulainen, 2002), hyperparameter optimization (Snoek et al., 2012; Loshchilov & Hutter, 2016; Jaderberg et al., 2017) and, neural architecture search (Zoph & Le, 2016; Real et al., 2017).”
DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, neutral): “Therefore, NAS is always formulated as a hyper-parameter optimization problem, whose algorithmic realization spans evolution algorithm [21, 7], reinforcement learning [27], Bayesian optimization [9], Monte Carlo Tree Search [24], and differentiable architecture search [14, 26, 3].”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “For instance, genetic algorithms (GA) have been applied to evolve increasingly complex neural network topologies and the weights simultaneously, in the so-called NeuroEvolution of Augmenting Topologies (NEAT) method [25, 35].”