Efficient natural evolution strategies

By Yi Sun, Daan Wierstra, Tom Schaul, J. Urgen Schmidhuber. 2009

130 citations recorded by the source corpus.

Efficient Natural Evolution Strategies (eNES) is a novel alternative to conventional evolutionary algorithms, using the natural gradient to adapt the mutation distribution. Unlike previous methods based on natural gradients, eNES uses a fast algorithm to calculate the inverse of the exact Fisher information matrix, thus increasing both robustness and performance of its evolution gradient estimation, even in higher dimensions. Additional novel aspects of eNES include optimal fitness baselines and importance mixing (a procedure for updating the population with very few fitness evaluations). The algorithm yields competitive results on both unimodal and multimodal benchmarks.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, methodology): “The version of ES we use in this work belongs to the class of natural evolution strategies (NES) [Wierstra et al., 2008, 2014, Yi et al., 2009, Sun et al., 2009, Glasmachers et al., 2010a,b, Schaul et al., 2011] and is closely related to the work of Sehnke et al. [2010].”
Parameter Space Noise for Exploration (2017, background): “Along the same lines, it was shown that the addition of parameter noise leads to better exploration by obtaining a policy that exhibits a larger variety of behaviors [8, 9].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “1 RL through Direct and Indirect Search in RNN Program Space It is possible to train small RNNs with a few 100 or 1000 weights using evolutionary algorithms [200, 255, 105, 56, 68] to search the space of NN weights [165, 307, 44, 321, 180, 259, 320, 164, 173, 69, 71, 187, 121, 313, 66, 270, 269, 305”
One Big Net For Everything (2018, background): “Without a teacher, reward-maximizing programs of an RNN must be learned through repeated trial and error, e.g., through artiﬁcial evolution [39, 107, 83, 40, 16, 18, 103, 14, 89, 88] [76, Sec.”
Evolution Strategies (2018, background): “…the complete covariance matrix of the mutation distribution (compare Section 2.4) are correlated mutations [3], the generating set adaptation [26], the covariance matrix adaptation ( CMA ) [27], a muta-tive invariant adaptation [28], and some instances of natural evolution strategies [29, 30, 31].”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “Natural evolutionary strategies(NES) (Wierstra et al, 2008; Sun et al, 2009) use all the generated solutions to estimate the gradient of the parameters of the generating function, and then use natural gradient ascent to improve these parameters.”
Exploring parameter space in reinforcement learning (2010, methodology): “Additional techniques were developed to enhance NES’ performance and viability, including importance mixing to reduce the number of required samples [24, 25], and exponential parametrization of the search distribution to guarantee invariance while at the same time providing an elegant and efficient ”
A Survey on Policy Search for Robotics (2013, methodology): “In contrast to the traditional gradient, the natural gradient avoids premature convergence on plateaus and overaggressive steps on steep ridges due to its isotropic convergence properties [3, 79].”
Exponential natural evolution strategies (2010, methodology): “Recent work on the Exact NES (eNES) algorithm [12, 11] has improved the robustness and reduced the computational complexity of NES.”
Stochastic search using the natural gradient (2009, methodology): “In a companion paper (Sun et al., 2009), we present an iterative method for computing F− which further reduces the time complexity from O ( d(4) ) to O ( d(3) ) .”
Limitations of the Empirical Fisher Approximation (2019, methodology): “Interestingly, there is also at least one example of an algorithm that was originally developed using the empirical Fisher and later found to work better with the Fisher [Wierstra et al., 2008, Sun et al., 2009].”
Intrinsically motivated neuroevolution for vision-based reinforcement learning (2011, methodology): “The particular instantiation of UL-ERL in this paper uses a computationally efficient variant of Natural Evolution Strate gies (NES; [5,23,24,26]), called Separable NES or SNES [17] as the evolutionary search method, and a competitive clustering algorithm based on Vector Quantization (VQ; [7]) for ”
Policy Optimization via Importance Sampling (2018, methodology): “[48] Yi Sun, Daan Wierstra, Tom Schaul, and Juergen Schmidhuber.”