The importance of experience replay database composition in deep reinforcement learning

By Tim de Bruin, Jens Kober, Karl Tuyls. 2015

88 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “It has been observed that large experience replay (Lin, 1992) can be detrimental to performance (de Bruin et al., 2015; Zhang & Sutton, 2017) and the diversity of states in the buffer is an important factor for performance (de Bruin et al., 2016).”
Selective Experience Replay for Lifelong Learning (2018, background): “Research shows that limited FIFO buffers can converge to homogeneous transitions when the system starts consistently succeeding, and that this lack of error samples can cause divergent behavior (de Bruin et al. 2015).”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “…that has been made in both a sequential multi-task setting [Isele and Cosgun, 2018, Rolnick et al., 2019], as well as in a single task setting [de Bruin et al., 2015, 2016, Zhang and Sutton, 2017, Wang and Ross, 2019], is that it is important to maintain a balance between the storage of new…”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “The diversity of states in the experience replay buffer is an important factor for performance [14] and our results indicate that 50 million experience tuples from a DQN agent are diverse enough to obtain good performance on most of the Atari 2600 games.”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “This issue persists even when the off-policy data comes from effective expert policies, which in principle should address any exploration challenge [7, 13, 12].”
Lenient Multi-Agent Deep Reinforcement Learning (2017, background): “Deep reinforcement learning agents have been shown to master numerous complex problem domains, ranging from computer games [15, 19, 25, 33] to robotics tasks [8, 10].”
Improved deep reinforcement learning for robotics through distribution-based experience retention (2016, background): “As reported previously [8], RL with deep neural network function approximators can fail when the experiences that are used to train the neural networks are not diverse enough.”
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning (2019, background): “It is worth noting that using the reply memory has two benefits [43].”
Continuous Deep Q-Learning with Model-based Acceleration (2016, unknown): “…have been applied to learn policies to play Atari games (Mnih et al., 2015; Schaul et al., 2015) and perform a wide variety of simulated and real-world robotic control tasks (Hafner & Riedmiller, 2011; Lillicrap et al., 2016; Levine & Koltun, 2013; de Bruin et al., 2015; Hafner & Riedmiller, 2011).”
An Optimistic Perspective on Offline Reinforcement Learning (2019, background): “We hypothesize that the size of the DQN Replay Dataset and its diversity (De Bruin et al., 2015) play a key role in the success of standard off-policy RL algorithms trained offline.”
Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past (2019, methodology): “In [4], replay buffers of different sizes were tested on DDPG, and result shows that a large enough buffer with enough data diversity can lead to better performance.”
Learning state representation for deep actor-critic control (2016, methodology): “The ERD is necessary to create such mini-batches, although care should be taken to keep the data within the database varied enough to prevent it from over-fitting [17].”