Efficient Learning and Planning Within the Dyna Framework

By Jing Peng, Ronald J. Williams. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Reinforcement Learning: A Survey (1996, neutral): “These problems are addressed by prioritized sweeping (Moore & Atkeson, 1993) and Queue-Dyna (Peng & Williams, 1993), which are two independently-developed but very similar techniques.”
Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping (2008, background): “The original papers on the Dyna architecture and most subsequent extensions (e.g., Singh 1992; Peng & Williams 1993; Moore & Atkeson 1993; Kuvayev & Sutton 1996) assumed a Markov environment with a tabular representation of states.”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, positive): “For instance Pan et al. (2018) and Goyal et al. (2018) use inverse dynamics models to effectively propagate value updates backwards in a manner similar to prioritized sweeping (Moore & Atke-son, 1993; Peng & Williams, 1993).”
Spatio-Temporal Abstractions in Reinforcement Learning Through Neural Encoding (2016, background): “In the early days of RL, understanding the behavior of agents was rather easy (Sutton, 1990); researchers focused on simpler problems (Peng and Williams, 1993), and policies were built using lighter models (Tesauro, 1994).”
Iterative Value-Aware Model Learning (2018, methodology): “These algorithmic differences are beside the empirical focus of Silver et al. [2017], Oh et al.”
Learning visual representations for perception-action systems (2011, background): “Learning requires about one or two hundred iterations to converge, which is not much by RL standards but difficult without simulated runs – a common practice in RL (Peng and Williams 1993).”
Locally Weighted Learning for Control (1997, background): “1), or as an incremental parallel process (Sutton, 1990; Moore and Atkeson, 1993; Peng and Williams, 1993).”
Kernel-Based Reinforcement Learning (2004, methodology): “Baird and Klopf (1993) apply the nearest neighbor algorithm to reinforcement learning, and Connell and Utgoff (1987), Peng and Williams (1995), and Atkeson, Moore, and Schaal (1997) apply locally weighted regression to physical control problems.”
Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, methodology): “The idea of backward search and prioritization were introduced together for tabular Dyna [Peng and Williams, 1993; Moore and Atkeson, 1993].”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, methodology): “Suppose that a dataset D = { s i , a i , s ′ i } i =1 with | D | tuples has been collected with a policy π D acting in the true (ground) MDP (Sutton, 1990; Moore & Atkeson, 1993; Peng & Williams, 1993).”
Learning to trade via direct reinforcement (2001, background): “In some problems, such as checkers [55], [56], backgammon [19], [20], navigating a maze [ 57 ], or”
Multitask reinforcement learning on the distribution of MDPs (2003, methodology): “For the former theme, that is about the size of each backup, Prioritized Sweeping (PS) was proposed.[3, 4] In the basic algorithm shown in Figure 5, simulated experiences are obtained by random backups.”
Problem solving with reinforcement learning (1995, methodology): “Once a world model has been learnt, it can also be used to perform value function updates oo-line (Sutton 1990, Peng and Williams 1993) or for planning ahead (Thrun and MM oller 1992).”
Structure-Adaptable Digital Neural Networks (1999, methodology): “A number of new algorithms exploiting this idea and optimizing the use of the internal model of the world have been proposed [11, 142, 151], but we have not analyzed if they are suitable for a digital implementation.”
Learning to Use Selective Attention and Short-Term Memory in Sequential Tasks (1996, background): “3If computational limitations or the number of states makes full value iteration too expensive, we can instead do Prioritized Sweeping [Moore and Atkeson, 1993], Q-DYNA [Peng and Williams, 1992] or even Q-learning [Watkins, 1989].”