Planning by Prioritized Sweeping with Small Backups
By Harm van Seijen, Richard S. Sutton. 2013
34 citations recorded by the source corpus.
Efficient planning plays a crucial role in model-based reinforcement learning. Traditionally, the main planning operation is a full backup based on the current estimates of the successor states. Consequently, its computation time is proportional to the number of successor states. In this paper, we introduce a new planning backup that uses only the current value of a single successor state and has a computation time independent of the number of successor states. This new backup, which we call a small backup, opens the door to a new class of model-based reinforcement learning methods that exhibit much finer control over their planning process than traditional methods. We empirically demonstrate that this increased flexibility allows for more efficient planning by showing that an implementation of prioritized sweeping based on small backups achieves a substantial performance improvement over classical implementations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Stefano Ermon
- Yoshua Bengio
- Jiaming Song
- Chelsea Finn
- Yuxi Li
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Sergey Levine
- Luc Van Gool
- Stefano Ermon
- Jiaming Song
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Dacheng Tao
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Stefano Ermon
- David Silver
- Jiaming Song
- Marc G. Bellemare
- Yuxi Li
- Chelsea Finn
Citations in the corpus
- Prioritized Experience Replay (2015, background): “The TD error provides one way to measure these priorities (van Seijen & Sutton, 2013).”
- Regret Minimization Experience Replay in Off-Policy Reinforcement Learning (2021, background): “In model-based planning, Prioritized Sweeping [13, 14, 15] selects the next state updates according to changes in value.”
- Experience Replay with Likelihood-free Importance Weights (2020, background): “In model-based planning, Prioritized Sweeping [25, 1, 41] selects the next state updates according to changes in value.”
- Knowledge Transfer for Deep Reinforcement Learning with Hierarchical Experience Replay (2017, methodology): “One common approach to measure these priorities is using the Temporal Difference (TD) error (Seijen and Sutton 2013).”
- Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay (2016, methodology): “The method proposes a prioritization regarding how often transitions are used for updates in the network based on the magnitude of their temporal difference (TD) error [24].”