Sample-based learning and search with permanent and transient memories

By David Silver, Richard S. Sutton, Martin Müller. 2008

138 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…is most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et al.,…”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “In Dyna-2 (Silver et al., 2008), the same process is applied but in the context the agent is currently in—starting the simulations from the current state—and adapting the policy locally (for example through separate fast weights).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “In Dyna-2 [34], the learning agent stores long-term and short-term memories, where a memory is deﬁned as the set of features and corresponding parameters used by an agent to estimate the value function.”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “This method was extended [33], [34] to work with deep neural network policies, but performed best with models that were not neural networks [9].”
Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, background): “Prior work has shown such linear value function approximations to be effective in the Monte-Carlo Search framework (Silver et al., 2008).”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, methodology): “Extensions of Dyna have been proposed (Silver et al., 2008; Sutton et al., 2012), including deep neural-networks as function approximations.”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “The most obvious is model-based RL, where it is used to train predictive models that can then be used for planning or policy learning [58, 53, 14, 13, 36, 22, 10, 63, 1].”
RTMBA: A Real-Time Model-Based Reinforcement Learning Architecture for robot control (2011, methodology): “DYNA-2 [7] extends DYNA to use UCT as its planning algorithm.”
Bayesian Learning of Recursively Factored Environments (2013, methodology): “Having access to a model also allows for hybrid techniques such as Dyna (Sutton, 1991; Silver et al., 2008) and TD-Search (Silver et al., 2012), which use a model of the environment to enhance the Proceedings of the 30 th International Conference on Machine Learning, Atlanta, Georgia, USA, 2013.”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, background): “Model-based RL agents attempt to build an environment model, e.g. Dyna-Q (Sutton, 1990), Dyna-2 (Silver et al., 2008).”
On the Analysis of Complex Backup Strategies in Monte Carlo Tree Search (2016, background): “Silver et al. applied the idea of eligibility traces to planning in Dyna2 (Silver et al., 2008).”
Learning and Using Models (2012, methodology): “UCT is also used as the planner inside several model-based reinforcement learning algorithms (Silver et al, 2008; Hester and Stone, 2010).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, methodology): “, 2018), feeding model-generated data into an otherwise model-free optimizer (Sutton, 1990; Silver et al., 2008; Lampe & Riedmiller, 2014; Kalweit & Boedecker, 2017; Luo et al., 2019), using model predictions to improve the quality of target values for temporal difference learning (Buckman et al.”