Agnostic System Identification for Monte Carlo Planning

By Erik Talvitie. 2015

18 citations recorded by the source corpus.

While model-based reinforcement learning is often studied under the assumption that a fully accurate model is contained within the model class, this is rarely true in practice. When the model class may be fundamentally limited, it can be difficult to obtain theoretical guarantees. Under some conditions the DAgger algorithm promises a policy nearly as good as the plan obtained from the most accurate model in the class, but only if the planning algorithm is near-optimal, which is also rarely the case in complex problems. This paper explores the interaction between DAgger and Monte Carlo planning, specifically showing that DAgger may perform poorly when coupled with a sub-optimal planner. A novel variation of DAgger specifically for use with Monte Carlo planning is derived and is shown to behave far better in some cases where DAgger fails.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Yoshua Bengio
3. Pieter Abbeel
4. Mohit Bansal
5. Jianfeng Gao
6. Ruslan Salakhutdinov
7. Joelle Pineau
8. Kyunghyun Cho

In Computational Theory and Mathematics

1. Sergey Levine
2. Yoshua Bengio
3. Nicolas Manfred Otto Heess
4. David Silver
5. Joelle Pineau
6. Peter W. Battaglia
7. Razvan Pascanu
8. S. M. Ali Eslami

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Nicolas Manfred Otto Heess
4. Yuxi Li
5. Timothy P. Lillicrap
6. Yoshua Bengio
7. Joelle Pineau
8. David Silver

In Probabilistic & Bayesian Methods

1. Yoshua Bengio
2. Oriol Vinyals
3. Danilo Jimenez Rezende
4. Nicolas Manfred Otto Heess
5. Razvan Pascanu
6. Karol Gregor
7. Max Welling
8. Joelle Pineau

Citations in the corpus

• Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Quite generally, it has been observed that model-based planning often leads to catastrophic outcomes given unavoidable imperfections of p when modelling complex environments (Talvitie, 2015).”
• Imagination-Augmented Agents for Deep Reinforcement Learning (2017, methodology): “In such domains, the performance of model-based agents employing standard planning methods usually suffers from model errors resulting from function approximation [18, 19].”
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “But for more complicated environments where the simulator is not exposed to the agent, the model-based RL usually suffers from the mismatch between the learned and real environments [12,28].”
• Self-Correcting Models for Model-Based Reinforcement Learning (2016, unknown): “As observed by Talvitie (2015), DAgger performs poorly in both versions, due to the suboptimal planner.”
• The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, result): “This theorem follows as a simple extension of existing results in the literature (Ross, 2013; Talvitie, 2015).”