Evolution Strategies as a Scalable Alternative to Reinforcement Learning

By Tim Salimans, Jonathan Ho, Xi Chen, Ilya Sutskever. 2017

1,787 citations recorded by the source corpus.

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Artificial Intelligence, Values, and Alignment (2020, methodology): “A third approach to alignment involves using evolutionary methods (Salimans et al. 2017).”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “The value-based algorithm we evaluate is the Deep Q-Network (DQN) (Mnih et al., 2013), and the policy-based algorithm is Evolutionary Strategies (ES) (Salimans et al., 2017).”
Recurrent World Models Facilitate Policy Evolution (2018, background): “Recent works [14, 26, 73, 94] have demonstrated that ES is a viable alternative to traditional Deep RL methods on many strong baselines.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Recent alternatives to asynchronous SGD for RL include using evolutionary processes (Salimans et al., 2017), distributed BA3C (Adamski et al.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “Asynchronous learning from parallel copies of the environment, as in A3C (Mnih et al. 2016), Gorila (Nair et al. 2015), or Evolution Strategies (Salimans et al. 2017), can be effective in speeding up learning, at least in terms of wall-clock time.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “The sophisticated nature of the RL alternative introduces complexity in its implementation: it requires backpropagation and poses challenges to parallelization (Salimans et al. 2017).”
Learning data augmentation policies using augmented random search (2018, background): “In concrete, it is 15 times more computationally efﬁcient than evolution strategy (ES) [8], which is the fastest competing method.”
Learning To Simulate (2018, background): “While alternatives for black-box optimization exist, like evolutionary algorithms (Salimans et al., 2017) or sampling-based methods (Bishop, 2006), we favor policy gradients in this work for their sample efﬁciency and success in prior art.”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Evolutionary search has been proposed as an interesting framework to learn both the structure as well as the connections in a neural network [16,17,18,19,20,21,22,23,24].”
World Models (2018, methodology): “Recent works (Fernando et al., 2017; Salimans et al., 2017; Ha, 2017b; Stanley & Clune, 2017) have conﬁrmed that ES is a viable alternative to traditional Deep RL methods on many strong baselines.”
A survey on Image Data Augmentation for Deep Learning (2019, neutral): “[111] directly compare evolutionary strategies with Reinforcement Learning.”
Noisy Networks for Exploration (2017, background): “Exploration in the policy space itself, for example, with evolutionary or black box algorithms [22, 9, 29], usually requires many prolonged interactions with the environment.”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, neutral): “Atari games are a common benchmark in deep reinforcement learning [94, 95, 117, 132].”
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, methodology): “Additionally, although previous implementations of SPSA for neural networks tend to sample v from Gaussian distributions (Salimans et al., 2017a; Ilyas et al., 2017) as is done in natural evolution strategies (Wierstra et al., 2008), we sample v from a Rademacher distribution ( i.e. , Bernoulli ± 1…”
Parameter Space Noise for Exploration (2017, methodology): “We use the numbers reported in [9] for ES and report the median return across three seeds for DQN.”
Ray: A Distributed Framework for Emerging AI Applications (2017, methodology): “We have already used this API to implement many state-of-the-art RL algorithms on top of Ray, including A3C [36], PPO [51], DQN [37], ES [49], DDPG [55], and Ape-X [27].”
Meta-Learning in Neural Networks: A Survey (2020, background): “EAs are attractive for several reasons [193]: (i) They can optimize any basemodel andmeta-objectivewith no differen-”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “perform hill-climbing in a subspace of policies [98], whilst more complex methods, such as compressed network search, impose additional inductive biases [53].”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, methodology): “Salimans et al. (2017) estimate gradients using a form of ﬁnite differences, evaluating hundreds of different parameter values in parallel to construct a gradient estimate.”