Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time

By Andrew W. Moore, Christopher G. Atkeson. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “ng gives equal importance to all transitions in the replay memory. A more sophisticated sampling strategy might emphasize transitions from which we can learn the most, similar to prioritized sweeping [17]. 4.1 Preprocessing and Model Architecture Working directly with raw Atari frames, which are 210”
Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, methodology): “We use prioritized sampling (inspired by Moore and Atkeson (1993)) to sample a fraction ρ of transitions from the higher priority pool and a fraction 1− ρ from the rest.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Experience replay allows to generalize prioritized sweeping (Moore & Atkeson, 1993) to the non-tabular setting (Schaul et al., 2015), and can also be used to simplify exploration by including expert (e.g., human) trajectories (Hester et al., 2017).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ow much a state-action pair contributes to the ﬁnal prediction. To focus on stateactions which are most relevant for learning is known from prioritized sweeping for model-based reinforcement learning [85]. Contribution analysis can be done by computing differences of return predictions when adding a”
Reinforcement learning in robotics: A survey (2013, methodology): “The idea of combining learning in simulation and in the real environment was popularized by the Dynaarchitecture (Sutton, 1990), prioritized sweeping (Moore and Atkeson, 1993), and incremental multi-step Q-learning (Peng and Williams, 1996) in reinforcement learning.”
Task-Free Continual Learning (2018, methodology): “A common DRL technique, known as “prioritized sweeping”, is to sample experiences with a large error more often than others [23].”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, background): “, 2018] to effectively propagate value updates backwards in a manner similar to prioritized sweeping [Moore and Atkeson, 1993; Peng and Williams, 1993].”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “which was in turn inspired by model-based RL algorithms like prioritized sweeping [112, 113].”
Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning (2018, methodology): “s 1,11,12), and introduce separated evaluation and target networks [16], which can help smooth the learning and avoid the divergence of parameters. Moreover, we leverage prioritized sampling strategy [18] to assist the framework learning from the most important historical transitions. 3.4 Offline Te”
An intrinsic reward mechanism for efficient exploration (2006, methodology): “A similar behavior is achieved with Prioritized Sweeping ( Moore & Atkeson, 1993 ) and Queue-Dyna (Peng & Williams, 1993) when a model is used to generate updates to the value function.”
Prioritized memory access explains planning and hippocampal replay (2017, unknown): “The AI literature suggests one alternative approach for prioritizing backups: Prioritized Sweeping (PS) [28, 27, 29].”
Representation Discovery for MDPs Using Bisimulation Metrics (2015, background): “…which allow computational effort to be focused on parts of the state space where changes are happening rapidly, similar to successful asynchronous or distributed dynamic programming techniques such as Bertsekas and Tsitsiklis (1996), Bertsekas and Castanon (1989), Moore and Atkeson (1993) .”
Convex Optimization: Algorithms and Complexity (2015, unknown): “Some examples of these methods are the Dyna architecture [Sutton, 1991] and prioritized sweeping [Moore and Atkeson, 1993].”
Exploiting Best-Match Equations for Efficient Reinforcement Learning (2011, methodology): “We propose an efficient best-match LVM algorithm that uses prioritized sweeping (Moore and Atkeson, 1993), a well-known technique for prioritizing model-based updates, to trade off extra computation for improved performance.”
Transfer Learning for Reinforcement Learning Domains: A Survey (2009, methodology): “A different problem formulation is posed by Perkins and Precup (1999) where the transition function, T , may change after reaching the goal.”
Fast Reinforcement Learning for Vision-guided Mobile Robots (2005, background): “Another alternative would be to apply Prioritized Sweeping (PS) [ 11 ], where a prioritized queue of state-action pairs is maintained in order to focus the search on transitions with big changes in Q. In our application, where the sample time is 0.125 seconds and the number of states is less than a ”
Adaptive Tile Coding for Value Function Approximation (2007, methodology): “Natural extensions to this work include testing it in higher-dimensional spaces, developing extensions that work with model-free RL, and using ideas from prioritized sweeping (Moore & Atkeson 1993) to speed learning by focusing updates on tiles affected by recent splits.”
Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty (1999, methodology): “Littman, & Moore, 1996) already used by several authors to favor exploration (Schmidhuber, 1991; Moore & Atkeson, 1993; Kaelbling, 1993; Koenig & Simmons, 1996).”