The Predictron: End-To-End Learning and Planning

By David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P. Reichert, Neil C. Rabinowitz, André Barreto and 1 more. 2016

310 citations recorded by the source corpus.

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple "imagined" planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “[41] David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel DulacArnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al.”
When to use parametric models in reinforcement learning? (2019, methodology): “One notable topic that is out of scope for this paper is the consideration of abstract models [Silver et al., 2017] and alternative ways to use these models in addition to classic planning [cf.”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “…of latent space models have been previously studied (Francois-Lavet et al., 2018; Zhang et al., 2018; Ha & Schmidhuber, 2018; Oh et al., 2017; Hafner et al., 2018; Kaiser et al., 2019; Silver et al., 2017), the parametrizations and training objectives used to learn such models have varied widely.”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Subsequent works have used RNN-based models to generate many frames into the future [8, 11, 25, 60], and also as an internal model to reason about the future [68, 90, 102].”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “The Predictron is another implicit planning network, trained in a supervised way directly from raw pixels, mimicking Bellman updates / iterations (Silver et al., 2016b).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “The predictron (Silver et al., 2017b) instead learns abstract-state transition functions in order to predict values.”
Value Prediction Network (2017, background): “VPN is close to and in-part inspired by Predictron [29] in that a recurrent neural network (RNN) acts as a transition function over abstract states.”
Toward Causal Representation Learning (2021, background): “Particularly relevant for causal leaning are generative world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see I NTRODUCTION above) to train RL agents [48], [99], [127], [178], [214], [231], [248], [268], [271].”
Learning to reinforcement learn (2016, background): “In particular, Ilin et al. (2007) showed that recurrent networks trained on random mazes can approximate dynamic programming procedures (see also Silver et al., 2017; Tamar et al., 2016).”
Towards Causal Representation Learning (2021, background): “world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see INTRODUCTION above) to train RL agents [127, 248, 98, 47, 271, 178, 232, 214, 268].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “But evidence of the effect of overshooting has been restricted to the agent’s performance evaluation [33, 31].”
World Models (2018, methodology): “Subsequent works have used RNN-based models to generate many frames into the future (Chiappa et al., 2017; Oh et al., 2015; Denton & Birodkar, 2017), and also as an internal model to reason about the future (Silver et al., 2016; Weber et al., 2017; Watters et al., 2017).”
Independently Controllable Features (2017, background): “Unlike recent work on the predictron (David Silver, 2017), our approach is not focused on solving a planning task, and the goal is simply to learn how the world works.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [67, 17, 54, 51, 59, 70], we choose to build upon Dyna-style approaches that optimize for a policy [64, 66, 73, 32, 26, 28, 44], specifically MBPO [29].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Prior methods have also explored incorporating computation that resembles model-based planning but without constraining the intermediate predictions of the planner to match plausible environment observations (Tamar et al., 2016; Racanière et al., 2017; Oh et al., 2017; Silver et al., 2017).”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, methodology): “To help policy training avoid local minima in more complex environments, we found it helpful to include auxiliary goal prediction tasks, similar to recent work in reinforcement learning (Dosovitskiy and Koltun 2016; Silver et al. 2016).”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “Tamar et al. [35], Silver et al. [36], and Oh et al. [37] all present neural networks whose architectures mimic classical iterative planning algorithms, and which are trained by reinforcement learning or to predict user-deﬁned, high-level features; in these, there is no explicit environment model.”
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics (2017, positive): “Progress in model-based Deep RL has thus far been limited, though methods like Embed to Control (Watter et al., 2015), Value Iteration Networks (Tamar et al., 2016), and the Predictron (Silver et al., 2016b) demonstrate the promise of this direction.”