Evolution-Guided Policy Gradient in Reinforcement Learning

By Shauharda Khadka, Kagan Tumer. Neural Information Processing Systems, 2018

298 citations recorded by the source corpus.

Deep Reinforcement Learning (DRL) algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically suffer from three core difficulties: temporal credit assignment with sparse rewards, lack of effective exploration, and brittle convergence properties that are extremely sensitive to hyperparameters. Collectively, these challenges severely limit the applicability of these approaches to real world problems. Evolutionary Algorithms (EAs), a class of black box optimization techniques inspired by natural evolution, are well suited to address each of these three challenges. However, EAs typically suffer from high sample complexity and struggle to solve problems that require optimization of a large number of parameters. In this paper, we introduce Evolutionary Reinforcement Learning (ERL), a hybrid algorithm that leverages the population of an EA to provide diversified data to train an RL agent, and reinserts the RL agent into the EA population periodically to inject gradient information into the EA. ERL inherits EA's ability of temporal credit assignment with a fitness metric, effective exploration with a diverse set of policies, and stability of a population-based approach and complements it with off-policy DRL's ability to leverage gradients for higher sample efficiency and faster learning. Experiments in a range of challenging continuous control benchmarks demonstrate that ERL significantly outperforms prior DRL and EA methods.

Subfields: Artificial Intelligence, Building and Construction

View Paper: https://www.semanticscholar.org/paper/d5805a80b63ed0a605e5469e321a7e3c42eaf324

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Building and Construction

Citations in the corpus

Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, unknown): “…(Jaderberg et al., 2017), to evolve intrinsic dense rewards (Jaderberg et al., 2018), to learn a neural loss for better generalization (Houthooft et al., 2018), to obtain diverse samples for faster off-policy learning (Khadka & Tumer, 2018), and to encourage exploration (Conti et al., 2018).”
Effective Diversity in Population-Based Reinforcement Learning (2020, methodology): “Population-based methods are particularly prominent in the Neuroevolution community (Stanley et al., 2019), but have recently been of increased interest in RL (Jung et al., 2020; Liu et al., 2019; Khadka and Tumer, 2018; Jung et al., 2020; Khadka et al., 2019; Pourchot and Sigaud, 2019).”
Evolving Reinforcement Learning Algorithms (2021, methodology): “For example, evolutionary algorithms are used to mutate the value or actor network weights (Whiteson & Stone, 2006; Khadka & Tumer, 2018), learn task reward (Faust et al., 2019), tune hyperparameters (Tang & Choromanski, 2020; Franke et al., 2020), or search for a neural network architecture (Song…”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, methodology): “Khadka & Tumer (2018) augment an off-policy RL method with diversiﬁed population information derived from an evolutionary algorithm.”