Learning model-based planning from scratch

By Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Manfred Otto Heess, Lars Buesing, Sébastien Racanière, David P. Reichert, Theophane Weber, Daan Wierstra, Peter W. Battaglia. 2017

102 citations recorded by the source corpus.

Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the "Imagination-based Planner", the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans. Before any action, it can perform a variable number of imagination steps, which involve proposing an imagined action and evaluating it with its model-based imagination. All imagined actions and outcomes are aggregated, iteratively, into a "plan context" which conditions future real and imagined actions. The agent can even decide how to imagine: testing out alternative imagined actions, chaining sequences of actions together, or building a more complex "imagination tree" by navigating flexibly among the previously imagined states using a learned policy. And our agent can learn to plan economically, jointly optimizing for external rewards and computational costs associated with using its imagination. We show that our architecture can learn to solve a challenging continuous control problem, and also learn elaborate planning strategies in a discrete maze-solving task. Our work opens a new direction toward learning the components of a model-based planning system and how to use them.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Risks from Learned Optimization in Advanced Machine Learning Systems (2019, methodology): “Reward-predictive model-based reinforcement learning might be one possible way of accomplishing this.[21] For example, an ML system could include a model directly trained to predict the base objective together with a powerful hard-coded optimization algorithm.”
Structured agents for physical construction (2019, methodology): “Using a learnt object-based model was recently put forward as an efficient method for planning in RL (Pascanu et al., 2017; Hamrick et al., 2017) and for predicting physical dynamics (Sanchez-Gonzalez et al.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “They have been used within both model-free (Wang et al., 2018b) and model-based (Hamrick et al., 2017; Pascanu et al., 2017; Sanchez-Gonzalez et al., 2018) continuous control, for model-free reinforcement learning (Hamrick et al., 2018; Zambaldi et al., 2018), and for more classical approaches to…”
Relational Deep Reinforcement Learning (2018, background): “While not a model in the technical sense, it is possible that the agent learns to exploit this relational architectural prior similarly to how an imagination-based agent’s forward model operates [35, 36, 37].”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “…2013; Lioutikov et al., 2014; Levine & Abbeel, 2014; Yip & Camarillo, 2014), mixture of Gaussians (Khansari-Zadeh & Billard, 2011), to neural networks (Hunt et al., 1992; Nagabandi et al., 2017; Kurutach et al., 2018; Tangkaratt et al., 2014; Sanchez-Gonzalez et al., 2018; Pascanu et al., 2017).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, neutral): “We refer to [Kurutach et al., 2018, Clavera et al., 2018, Sun et al., 2018b, Chua et al., 2018, Sanchez-Gonzalez et al., 2018, Pascanu et al., 2017, Khansari-Zadeh and Billard, 2011, Luo et al., 2018] and the reference therein for recent work on model-based RL. Off-policy reinforcement learning…”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “[52] extend this work by focusing on explicit planning in sequential environments, and learn how to construct a plan iteratively.”
Graph networks as learnable physics engines for inference and control (2018, methodology): “Our approach may also be useful within imagination-based planning frameworks (Hamrick et al., 2017; Pascanu et al., 2017), as well as integrated architectures with GN-like policies (Wang et al.”
A Brief Survey of Deep Reinforcement Learning (2017, methodology): “using representation learning to create distributed embeddings is a particular strength of DRL, and has been successfully utilised for other purposes [161, 100].”
Relational Forward Models for Multi-Agent Learning (2018, positive): “Here we show that, by embedding RFM modules in RL agents, they can learn to coordinate with one another faster than baseline agents, analogous to imagination-augmented agents in single-agent RL settings (Hamrick et al., 2017; Pascanu et al., 2017; Weber et al., 2017).”
Learning to Search with MCTSnets (2018, neutral): “Pascanu et al. (2017) also investigate learning-to-plan using neural networks.”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, background): “They often learn a policy based on interaction networks’ rollouts (Racani ` ere et al., 2017; Hamrick et al., 2017; Pascanu et al., 2017).”
Differentiable MPC for End-to-end Planning and Control (2018, background): “To tackle continuous state and action spaces, Pascanu et al. [2017] propose a neural architecture which uses an abstract environmental model to plan and is trained directly from an external task loss.”
Model-Based Planning with Discrete and Continuous Actions (2017, methodology): “We used a spaceship domain inspired by recent works (Hamrick et al., 2017; Pascanu et al., 2017), where the agent must pilot a spaceship in the presence of planets and their gravitational forces.”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “In particular, many model-based agents learn a model not only for the state transitions (which is safe), but also a model for the rewards from the observed reward signal (e.g. Ha and Schmidhuber, 2018; Kaiser et al., 2019; Pascanu et al., 2017; Weber et al., 2017).”
Deep Reinforcement Learning for Cyber Security (2019, methodology): “Alternatively, model-based and model-free combination approaches, such as model-free policy with planning capabilities [166], [167] or model-based lookahead search [30], can be used as they aggregate advantages of both methods.”
Predictor-Corrector Policy Optimization (2018, background): “Learning-to-plan attempts to train the planning process end-to-end [13, 14], so the performance of a given planning structure is directly optimized.”