Imagination-Augmented Agents for Deep Reinforcement Learning

By Sébastien Racanière, Theophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Manfred Otto Heess, Yujia Li and 5 more. 2017

599 citations recorded by the source corpus.

We introduce Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects. In contrast to most existing model-based reinforcement learning and planning methods, which prescribe how a model should be used to arrive at a policy, I2As learn to interpret predictions from a learned environment model to construct implicit plans in arbitrary ways, by using the predictions as additional context in deep policy networks. I2As show improved data efficiency, performance, and robustness to model misspecification compared to several baselines.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “One notable topic that is out of scope for this paper is the consideration of abstract models [Silver et al., 2017] and alternative ways to use these models in addition to classic planning [cf. Weber et al., 2017].”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “, 2018), combining the benefits of model-based and model-free approaches (Kalweit & Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha & Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video predictio”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “reason about the future using this model. Subsequent works have used RNN-based models to generate many frames into the future [8, 11, 25, 60], and also as an internal model to reason about the future [68, 90, 102]. In this work, we used evolution strategies (ES) to train our controller, as this offe”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “Finally, learned environment models, such as those used for conditional video prediction, are lever-aged in Weber et al. (2017).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “Imagination-augmented agents (Weber et al., 2017) learn to improve policies by aggregating rollouts predicted by a model.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “ localization, where robots are actively guided to investigate unfamiliar regions by humans [29, 58]. Another direction in this area is to learn a world model [4, 37, 36] that generates latent states [88, 38, 77]. If the dynamics of the world are also learned, then it can be applied to planning [65,”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “…resulting model may be used in various ways, e.g. for planning (Oh et al., 2017; Schrittwieser et al., 2019), generation of synthetic training data (Weber et al., 2017), better credit assignment (Heess et al., 2015), learning useful internal representations and belief states (Gregor et al., 2019;…”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “As such, it would be interesting to use it in a reinforcement learning setting in conjunction with Imagination-Augmented Agents (Weber et al., 2017) or more generally as a world model (Ha and Schmidhuber, 2018), especially for settings with simple backgrounds, e. g., games like Montezuma’s Revenge o”
World Models (2018, methodology): “, 2015; Denton & Birodkar, 2017), and also as an internal model to reason about the future (Silver et al., 2016; Weber et al., 2017; Watters et al., 2017).”
Relational Deep Reinforcement Learning (2018, background): “While not a model in the technical sense, it is possible that the agent learns to exploit this relational architectural prior similarly to how an imagination-based agent’s forward model operates [35, 36, 37].”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Prior methods have also explored incorporating computation that resembles model-based planning but without constraining the intermediate predictions of the planner to match plausible environment observations (Tamar et al., 2016; Racanière et al., 2017; Oh et al., 2017; Silver et al., 2017).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “, 2020), with the rollout in latent space sometimes referred to as ‘imagination’ (Racanière et al., 2017; Hafner et al., 2019).”
Relational recurrent neural networks (2018, methodology): “Mini Pacman with viewport We follow the formulation of Mini Pacman from [23].”