Multi-Step Dyna Planning for Policy Evaluation and Control

By Hengshuai Yao, Richard S. Sutton, Shalabh Bhatnagar, Diao Dongcui, Csaba Szepesvari. 2009

23 citations recorded by the source corpus.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Yuxi Li
5. Ofir Nachum
6. Tianhe Yu
7. Honglak Lee
8. Satinder Singh

Citations in the corpus

• Value Prediction Network (2017, methodology): “Dyna-Q [32, 34, 39] integrates model-free and modelbased RL by learning an observation-prediction model and using it to generate samples for Q-learning in addition to the model-free samples obtained by acting in the real environment.”
• MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], specifically MBPO [28].”
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, background): “Linear Dyna was later extended to incorporate multi-step projections of sampled features, which was found to speed learning (Yao et al., 2009).”
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “In order to combat this issue, RL researchers are actively working on combining model-free and model-based RL [33,34,35,36].”
• Reinforcement learning with misspecified model classes (2013, background): “Model-based (MB) RL techniques have been shown to be more sample-efficient than model-free RL methods [4], [5], [6], [7], making them a natural choice for tackling real-world robotics problems [8].”