Dyna, an integrated architecture for learning, planning, and reacting

By Richard S. Sutton. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Dynamic programming methods, such as Q-learning algorithms, in principle can offer a more attractive option for offline reinforcement learning as compared to pure policy gradients.”
Structured agents for physical construction (2019, methodology): “Here we combine our DQN agent with MCTS, in the spirit of Sutton (1991) and Azizzadenesheli et al. (2018).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “There also exists another branch of model-based algorithms where a model is learned from experiences. with which the agent can interact and collect imaginary rollouts (Sutton, 1991), It has also been extended with DRL methods (Weber et al., 2017; Kalweit and Boedecker, 2017).”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, methodology): “The classic Dyna [32] algorithm proposed to use a model to generate simulated experience that could be included in a model-free algorithm.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “In Proceedings of the International Workshop on Synergistic Intelligence Dynamics.”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, methodology): “Combining model-free and model-based methods has been investigated in a number of prior works [21].”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “While any RL or planning algorithm can be used to learn the optimal policy for (cid:99) M , we focus speciﬁcally on MBPO [20, 57] which was used in MOPO.”
Reinforcement Learning with Augmented Data (2020, background): “While usually not viewed as a data augmentation technique, the idea of generating fake or synthetic rollouts to improve the data-efﬁciency of RL agents has been proposed in the Dyna framework [32].”
Benchmarking Model-Based Reinforcement Learning (2019, methodology): “In the Dyna algorithm [45, 46, 47], training iterates between two steps.”
The Option-Critic Architecture (2016, methodology): “Classical dynamic programming methods such as value iteration and policy iteration [Puterman, 1994] or Dyna planning [Sutton, 1991; Sorg and Singh, 2010] can be used to provide a solution.”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “The trained model can be used as a simulator in which the policy can be trained, and also provides gradient information (Sutton, 1990; Deisenroth & Rasmussen, 2011; Depeweg et al., 2017; Sutton, 1991).”
A Deeper Look at Experience Replay (2017, methodology): “However, the key difference is that Dyna only samples states , while experience replay samples full transitions , which may be biased and potentially harmful.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Second, model-assisted MF approaches use the dynamics models to augment the real environment data by imagining policy roll-outs [33, 29, 34, 22].”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “Model-Based RL There are many types of model-based RL algorithms (Sutton, 1991; Deisenroth & Rasmussen, 2011; Heess et al., 2015).”
Learning model-based planning from scratch (2017, methodology): “The classic “Dyna” algorithm learns a model, which is then used to train a policy [21].”
Exploring Model-based Planning with Policy Networks (2019, background): “Dyna [38, 39] algorithm alternately performs sampling in the real environments and optimize the controllers on the learned model of the environments.”