Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains

By Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, Martha White. 2018

49 citations recorded by the source corpus.

Model-based strategies for control are critical to obtain sample efficient learning. Dyna is a planning paradigm that naturally interleaves learning and planning, by simulating one-step experience to update the action-value function. This elegant planning strategy has been mostly explored in the tabular setting. The aim of this paper is to revisit sample-based planning, in stochastic and continuous domains with learned models. We first highlight the flexibility afforded by a model over Experience Replay (ER). Replay-based methods can be seen as stochastic planning methods that repeatedly sample from a buffer of recent agent-environment interactions and perform updates to improve data efficiency. We show that a model, as opposed to a replay buffer, is particularly useful for specifying which states to sample from during planning, such as predecessor states that propagate information in reverse from a state more quickly. We introduce a semi-parametric model learning approach, called Reweighted Experience Models (REMs), that makes it simple to sample next states or predecessors. We demonstrate that REM-Dyna exhibits similar advantages over replay-based methods in learning in continuous state problems, and that the performance gap grows when moving to stochastic domains, of increasing size.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, background): “…one could apply a nearest-neighbours or kernel approach to a replay buffer, and thereby obtain a non-parametric model that can be equivalent to replay when sampled at the observed states, but that can interpolate and generalise to unseen states when sampled at other states [Pan et al., 2018].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Interestingly, the close connection between experience replay and planning was also recently highlighted in the RL literature (Pan et al., 2018; Eysenbach, Salakhutdinov, and Levine, 2019).”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “…efficient than model-based approaches (Van Hasselt et al., 2019), and using the DQN Replay Dataset on Atari 2600 games for designing non-parametric replay models (Pan et al., 2018) and parametric world models (Kaiser et al., 2019) is another promising direction for improving sampleefficiency in RL.”
Simplifying Deep Reinforcement Learning via Self-Supervision (2021, background): “While many studies have been focused on advancing experience replay (Andrychow-icz et al. 2017; Pan et al. 2018; Zhang and Sutton 2017; No-vati and Koumoutsakos 2018; Zha et al. 2019b), the main goal of the prior work is to accelerate or stabilize the training of value functions.”
Planning with Expectation Models (2019, background): “The efficiency of planning can be significantly improved with clever search control strategy such as prioritized sweeping [Moore and Atkeson, 1993; Sutton et al., 2012; Pan et al., 2018].”
Revisiting Fundamentals of Experience Replay (2020, background): “Beyond direct manipulation of these properties of the buffer, improving and understanding experience replay algorithms remains an active area of research (Pan et al., 2018; Schlegel et al., 2019; Zha et al., 2019; Novati & Koumoutsakos, 2019; Sun et al., 2020; Lee et al., 2019).”