Prioritized Experience Replay
By Tom Schaul, John Quan, Ioannis Antonoglou, David Silver. 2015
4,495 citations recorded by the source corpus.
Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Chelsea Finn
In Cognitive Neuroscience
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Yoshua Bengio
- Jeff Clune
- Marc G. Bellemare
- Timothy P. Lillicrap
- Matthew Botvinick
In Electrical and Electronic Engineering
- Sergey Levine
- Ofir Nachum
- Shimon Whiteson
- Pieter Abbeel
- Yuxi Li
- Shixiang Shane Gu
- Mohammad Norouzi
- Aviral Kumar
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Yuxi Li
- Shixiang Shane Gu
- Marc G. Bellemare
Citations in the corpus
- What can the brain teach us about building artificial intelligence? (2016, background): “More recent variants of the DQN have demonstrated superior performance (Schaul et al., 2015; Stadie et al., 2016; van Hasselt, Guez, & Silver, 2016), and the current best network with smarter replay and other improvements now achieves about 83% of the professional gamer’s score (Schaul et al.”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “We use the Atari domain to compare against state of the art results (Van Hasselt et al., 2015; Wang et al., 2015; Schaul et al., 2015; Nair et al., 2015; Mnih et al., 2015), as well as to carry out a detailed stability and scalability analysis of the proposed methods.”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “ always kept in the buffer. Transitions are sampled for learning with probability proportional to a priority, computed from their TD error at the moment they are added to and sampled from the buffer (Schaul et al., 2015). The training objective for the agent’s policy is the the cost function J(Q) = ”
- Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “For this experiment, we also made use of prioritized experience replay (Schaul et al., 2015) to allow the model to more frequently learn from relatively rare valid samples.”
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Models were trained using prioritized experience replay [88], where episode priority was tuned to encourage replay buffer training data for each skill to be close to 50% success.”
- Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “[35] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.”
- Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “, 2017), along with multi-step returns and prioritized experience replay (Schaul et al., 2016; Horgan et al., 2018), and distributional methods (Bellemare et al.”
- The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “The algorithms that we compare Reactor to are: DQN (Mnih et al., 2015), Double DQN (Van Has-selt et al., 2016), DQN with prioritized experience replay (Schaul et al., 2015), duelling architecture and prioritized duelling architecture (Wang et al., 2015).”
- Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “The softmax consistency leads us to develop Path Consistency Learning (PCL), an RL algorithm that has a flavor similar to both actor-critic in that it maintains and jointly learns a model of the state values and a model of the policy, and Q-learning in that it minimizes a measure of temporal inconsi”
- PGQ: Combining policy gradient and Q-learning (2016, background): “Furthermore, the replay buffer can be used to store and replay ‘important’ past experiences by prioritizing those samples (Schaul et al., 2015).”
- Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Prioritized experience replay has the potential to provide more efficient learning compared to uniform experience replay (Schaul et al., 2015; Horgan et al., 2018).”
- When to use parametric models in reinforcement learning? (2019, background): “, 2016], prioritised experience replay [Schaul et al., 2016], noisy networks for exploration [Fortunato et al.”
- Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “Both buffers store 1-step and 10-step transitions and are prioritized [18].”
- Distributed Prioritized Experience Replay (2018, methodology): “In this paper we describe an approach to scaling up deep reinforcement learning by generating more data and selecting from it in a prioritized fashion (Schaul et al., 2016).”
- Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “As a proxy for learning potential, prioritized experience replay (Schaul et al. 2015) samples transitions with probability pt relative to the last encountered absolute TD error:”
- Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “While some have reported results in the ALE using learning curves (e.g., Mnih et al. 2016; Ostrovski, Bellemare, van den Oord, & Munos 2017; Schaul et al. 2016), it is difficult to even effectively display, let alone comprehend and compare, 60 learning curves.”
- Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “The overall objective is to maximise total performance across all these auxiliary tasks, argmax θ Eπ[R1:∞] + λc ∑ c∈C Eπc [R (c) 1:∞], (1) where, R(c)t:t+n = ∑n k=1 γ kr (c) t is the discounted return for auxiliary reward r (c), and θ is the set of parameters of π and all π(c)’s.”
- Dueling Network Architectures for Deep Reinforcement Learning (2015, methodology): “The results of Schaul et al. (2016) are the current published state-of-the-art.”
- Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “Retrace( λ ) can learn from full returns retrieved from past policy data, as in the context of experience replay (Lin, 1993), which has returned to favour with advances in deep reinforcement learning (Mnih et al., 2015; Schaul et al., 2016).”