An investigation of model-free planning

By Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles and 3 more. 2019

130 citations recorded by the source corpus.

The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the characteristics typically associated with a model-based planner. We measure our agent's effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has many of the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-of-the-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “We use the same stochastic version of MiniPacman as released by Guez et al. (2019).”
Planning with Goal-Conditioned Policies (2019, methodology): “Lastly, we note that while a number of papers have studied how to combine model-free and modelbased methods [54, 41, 23, 56, 44, 51, 39], our method is substantially different from these approaches: we study how to use model-free policies as the abstraction for planning, rather than using models [54”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “In contrast to approaches based on iterative computations performed in activation space (Guez et al., 2019; Dehghani et al., 2018; Graves, 2016; Schrittwieser et al., 2020), humans can directly supervise iterative reasoning steps in natural language.”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, methodology): “We also evaluate ITER on Boxoban (Guez et al., 2018; Schrader, 2018) 1 .”
The alignment problem from a deep learning perspective (2022, background): “Guez et al. [2019] showed evidence that implicit goal-directed planning can emerge in sequential decision-making models, and can generalize to problems harder than those seen during training.”
Rapid Task-Solving in Novel Environments (2020, background): “The effectiveness of learned, implicit planning was demonstrated further in fully-observed spatial environments using grid-structured memory and convolutional update functions (Tamar et al., 2016; Lee et al., 2018b; Guez et al., 2019).”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, methodology): “We procedurally generate levels as in (Guez et al., 2019; Racanière et al., 2017).”