Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

By Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, Jeff Clune. 2017

382 citations recorded by the source corpus.

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “Prior works have used evolutionarymethods to evolve neural networks [8, 16, 34, 38, 45].”
Transfer Learning with Neural AutoML (2018, background): “An emerging body of neuro-evolution research has adapted genetic algorithms for these complex optimization problems [20], including to set the parameters of existing deep networks [21], evolve image classifiers [5], and evolve generic deep neural networks [6].”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “In an companion paper (Conti et al., 2017), we also demonstrate a similar ﬁnding, by hybridizing novelty search with ES to create NS-ES, and show that it too can help deep neural networks avoid deception in challenging RL benchmark domains.”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, background): “This is key for acquiring world models and repertoires of parameterized skills [3, 4, 5], to efficiently bootstrap exploration for deep reinforcement learning problems with rare or deceptive rewards [6, 7], or to quickly repair strategies in case of damages [8].”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “Overall, ES has exhibited performance on par with some of the traditional, simple gradient-based RL algorithms on difﬁcult RL domains (e.g. DQN [7] and A3C [60]), including Atari environments and simulated robot locomotion [47, 61].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “IM is often motivated by psychological concepts such as curiosity [12, 13] or novelty-seeking [7, 14], which play a role in how humans explore and learn.”
State Representation Learning for Control: An Overview (2018, methodology): “ES optimization methods have shown to be eﬃcient for deep reinforcement learning [Conti et al., 2017] but they could then take a clear advantage of a lower dimension input to explore faster the parameter space than using raw data [Alvernaz and Togelius, 2017].”
Learning Novel Policies For Tasks (2019, methodology): “Some research uses the approach of curiosity-driven exploration (Schmidhuber, 1991; Sun et al., 2011; Conti et al., 2018) and applies these techniques to reinforcement learning (Houthooft et al., 2016; Hester & Stone, 2017).”
Diversity-Driven Exploration Strategy for Deep Reinforcement Learning (2018, background): “A work parallel to ours similarly employs novelty search for exploration [19].”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “Many have shown such a combination is effective, whether in a weighted combination of novelty and reward [30], multi-objective algorithms [115, 117], or algorithms that intelligently rebalance between novelty and performance as a function of learning progress [30].”
Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods (2020, methodology): “…Mnih et al. 2016; Ecoffet et al. 2019; Nair et al. 2015; Horgan et al. 2018; Jader-berg et al. 2018; Sutton and Barto 2018) and population-based methods (Salimans et al. 2017; Jaderberg et al. 2017; Such et al. 2017; Conti et al. 2018; Wang et al. 2019; Stanley et al. 2019; Cully et al. 2015).”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, positive): “Attention-Based Policy Architecture: Attention mechanism is widely used in RL policy representation to capture object level information (Duan et al., 2017; Wang et al., 2018), represent relations (Zambaldi et al., 2018; Malysheva et al., 2018; Yang et al., 2019) and extract communication channels (J”
Malthusian Reinforcement Learning (2018, neutral): “vations for single-agent exploration [2, 3, 9, 25, 29].”
Accelerating Reinforcement Learning through GPU Atari Emulation (2019, methodology): “Although the wall clock metric only partially describes the efficiency of the learning procedure, it is among the most important from the practical point of view [18, 21, 5]; furthermore, reducing the turnaround time also allows to speed up the research of sample efficient algorithms as well.”
Data-efficient neuroevolution with kernel-based surrogate models (2018, background): “h the same performance by several times. scale [22, 34]. With a body of existing NE research, on topics such as exploration and overcoming deception, already being leveraged on deep learning problems [7, 31], NE is poised for a surge in interest. The main challenge for deploying NE techniques in man”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[40] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O.”