Competitive Experience Replay

By Hao Liu, Alexander R. Trott, Richard Socher, Caiming Xiong. 2019

60 citations recorded by the source corpus.

Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Model-augmented Prioritized Experience Replay (2022, methodology): “Then the loss for C θ is where L R θ and L T θ are the losses for R θ and T θ , respectively, with adaptively changing positive coefficients ξ 1 , ξ 2 , and ξ 3 by employing a dynamic method in (Liang & Zhang, 2020; Liu et al., 2019b).”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “learning [27], dynamic goals [14], or they adapt goal rela-”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “…incorporate recent advances to our implementation such as exploration with State Marginal Matching (Lee et al., 2019), vector quantization techniques to impose discrete priors in VAEs (van den Oord et al., 2017), and relabeling techniques to optimize deceptive reward functions (Trott et al., 2019).”
Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, methodology): “These limitations have motivated the recent development of methods to enable learning from sparse rewards (Schulman et al., 2017; Liu et al., 2019), methods to learn latent representations that facilitate shaped reward (Ghosh et al., 2018; Nair et al., 2018; Warde-Farley et al., 2019), and learning…”
Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following (2019, background): “These extensions deal with automatic curriculum learning (Liu et al., 2019), dynamic goals (Fang et al.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “The maze environments are adapted from the open-source implementation 3 by Trott et al. (2019).”