The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces

By G. Holland, Erik Talvitie, Michael H. Bowling, Zach Holland. 2018

44 citations recorded by the source corpus.

Dyna is a fundamental approach to model-based reinforcement learning (MBRL) that interleaves planning, acting, and learning in an online setting. In the most typical application of Dyna, the dynamics model is used to generate one-step transitions from selected start states from the agent's history, which are used to update the agent's value function or policy as if they were real experiences. In this work, one-step Dyna was applied to several games from the Arcade Learning Environment (ALE). We found that the model-based updates offered surprisingly little benefit over simply performing more updates with the agent's existing experience, even when using a perfect model. We hypothesize that to get the most from planning, the model must be used to generate unfamiliar experience. To test this, we experimented with the "shape" of planning in multiple different concrete instantiations of Dyna, performing fewer, longer rollouts, rather than many short rollouts. We found that planning shape has a profound impact on the efficacy of Dyna for both perfect and learned models. In addition to these findings regarding Dyna in general, our results represent, to our knowledge, the first time that a learned dynamics model has been successfully used for planning in the ALE, suggesting that Dyna may be a viable approach to MBRL in the ALE and other high-dimensional problems.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, methodology): “Using six Atari games as a benchmark Holland et al. (2018) measure the impact of planning shapes on performance of the Dyna-DQN algorithm and include ablations comparing scores obtained with perfect and imperfect models.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “[21], in fact, observe that it might require to make the depth up to the effective horizon in order to draw an improvement.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “While ground-truth models are most effective when queried for long horizons (Holland et al., 2018), inaccuracies in learned models tend to make long rollouts unreliable.”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, background): “(Holland et al., 2018) points out that incorporating long term future by doing Dyna style planning could be useful for model based RL.”
Model Primitive Hierarchical Lifelong Reinforcement Learning (2019, background): “The world however is complex, and learning models that are consistent enough to plan with is not only hard [24], but planning with such one-step models is suboptimal [11].”
Do recent advancements in model-based deep reinforcement learning really improve data efficiency? (2019, methodology): “Holland et al. (2018) explored the performance of the model-based approach given either perfect model, model pretrained on expert data (pretrained model), or model learned alongside the agent’s value function (online model).”