A Deeper Look at Planning as Learning from Replay
By Harm Vanseijen, Richard S. Sutton. 2015
59 citations recorded by the source corpus.
Subfields: Reinforcement Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Sergey Levine
- Aviral Kumar
- George Tucker
- Justin Fu
- Ofir Nachum
- Shixiang Shane Gu
- Chelsea Finn
- Pieter Abbeel
In Meta & Continual Learning
- Martha White
- Aviral Kumar
- George Tucker
- Somjit Nath
- Ajin George Joseph
- Sergey Levine
- Aaron C. Courville
- Adam White
Citations in the corpus
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “In the linear function approximation case, it is known that modelbased updates and fitted value iteration updates actually produce identical iterates (Vanseijen and Sutton, 2015; Parr et al., 2008), though it is unknown whether this relationship holds under non-linear function approximation.”
- A Brief Survey of Deep Reinforcement Learning (2017, neutral): “actually be considered a simple model [128].”
- The Primacy Bias in Deep Reinforcement Learning (2022, background): “We conjecture that a model-based perspective can offer an explanation: since the replay buffer can be seen as a non-parametric model of the world (Vanseijen & Sutton, 2015; van Hasselt et al., 2019b), after a reset, the agent forgets the behaviors learned in the past while preserving its model in…”
- Two-Timescale Networks for Nonlinear Value Function Approximation (2019, methodology): “We compare several choices: TD, least-squares TD (LSTD) (Bradtke and Barto, 1996), forgetful LSTD (FLSTD) (van Seijen and Sutton, 2015), emphatic TD (Sutton et al., 2016), gradient TD (the TDC variant) (Sutton et al., 2009) and their true-online versions (van Seijen and Sutton, 2014; van Hasselt et…”
- Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, background): “Note that [van Seijen and Sutton, 2015] relate Dyna and ER, but specifically for a theoretical equivalence in policy evaluation based on a non-standard form of replay related to true online methods, and thus we do not include it.”