GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

By Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. 2017

169 citations recorded by the source corpus.

In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at this http URL.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Decoupling exploration and exploitation has been previously attempted by achieving sufﬁcient diversity in data in an exploratory phase (Colas et al., 2018).”
Continual Learning for Robotics (2019, background): “This is generally done by a reward function that deﬁnes credit assignment [106], or it can also be deﬁned internally by more abstract rules such as self-supervision [54, 152], intrinsic motivation or curiosity [113, 142] as in [46, 27, 30, 78].”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, background): “This is key for acquiring world models and repertoires of parameterized skills [3, 4, 5], to efficiently bootstrap exploration for deep reinforcement learning problems with rare or deceptive rewards [6, 7], or to quickly repair strategies in case of damages [8].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, positive): “[86]), in which exploration follows a reward-agnostic Goal Exploration Process [87] (an algorithm similar to novelty search [7]), from which experience is collected to prefill the replay buffer of an off-policy RL algorithm, in this case DDPG [88].”
Evolution-Guided Policy Gradient in Reinforcement Learning (2018, methodology): “used an evolutionary method (Goal Exploration Process) to generate diverse samples followed by a policy gradient method for fine-tuning the policy parameters [7].”
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, methodology): “Also, using a population-based IMGEP architecture can help bootstrap a deep RL agent (Colas et al., 2018).”
CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning (2018, background): “Off-policy learning enables the use of any transition to improve the current policy: transitions collected from a different version of the current policy [10], from a population of exploratory policies [11], or even by demonstrations [12].”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “There has also been effort putting to either explicitly or implicitly separate exploration and exploitation (Colas et al., 2018; Forestier et al., 2017; Levine et al., 2016).”
The problem with DDPG: understanding failures in deterministic environments with sparse rewards (2019, unknown): “More related to our work, several papers have studied failure to gather rewarded experience from the environment due to poor exploration (Colas et al., 2018; Fortunato et al., 2017; Plappert et al., 2017), but we go beyond this issue by studying a case where the reward is actually found but not…”
Learning Novel Policies For Tasks (2019, background): “GEP-PG (Colas et al., 2018) tried to decouple exploration and exploitation by focus in one or the other in stages to encourage exploration.”
Multi-Stage Episodic Control for Strategic Exploration in Text Games (2022, unknown): “…control differs from prior approaches that add exploration biases to a single policy through curiosity bonuses (Pathak et al., 2017; Tang et al., 2017) or use different reward functions to train a separate exploration policy (Colas et al., 2018; Sch ¨ afer et al., 2021; Whitney et al., 2021).”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “[115] split the exploration into two phases: (i) explore randomly and (ii) compare experiences to an expert-created imitation to determine the good behavior.”
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning (2018, unknown): “, 2015), from a population of exploratory policies (Colas et al., 2018), or even from demonstrations (Večerı́k et al.”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Other methods use data collected from a randomly initialized policy or heuristics based on data collected online to design a non-parametric [11, 42, 14] or parametric [33, 29] goal distribution.”