Shaping Belief States with Generative Environment Models for RL

By Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, Aäron van den Oord. 2019

124 citations recorded by the source corpus.

When agents interact with a complex environment, they must form and maintain beliefs about the relevant aspects of that environment. We propose a way to efficiently train expressive generative models in complex environments. We show that a predictive algorithm with an expressive generative model can form stable belief-states in visually rich and dynamic 3D environments. More precisely, we show that the learned representation captures the layout of the environment as well as the position and orientation of the agent. Our experiments show that the model substantially improves data-efficiency on a number of reinforcement learning (RL) tasks compared with strong model-free baseline agents. We find that predicting multiple steps into the future (overshooting), in combination with an expressive generative model, is critical for stable representations to emerge. In practice, using expressive generative models in RL is computationally expensive and we propose a scheme to reduce this computational burden, allowing us to build agents that are competitive with model-free baselines.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “See also DeepMind’s MONet (Burgess et al., 2019)and MERLIN (Wayne et al., 2018) as well the expressive generative model in (Gregor et al., 2019)] and. e.g., inverse graphics papers by Vicarious (Kansky et al., 2017; George et al., 2017) and Josh Tenenbaum’s group (Mao et al., 2019; Veerapaneni et…”
Towards Causal Representation Learning (2021, background): “However, many of the current approaches (regardless of structure), only build partial models of the environment [88].”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “The setup for these experiments is similar to (Gregor et al., 2019), where the agent is trained using the IMPALA al-gorithm (Espeholt et al., 2018), and the model is trained alongside the agent via ELBO optimization on the data collected by the agent.”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “…suitability to un-supervised training, for example, to improve sample efﬁciency in reinforcement learning (Ha and Schmidhuber [2], Racanière et al. [3]), to synthesise training instances for data augmentation (Lee et al. [4]) or to capture beliefs about the state of a scene (Gregor et al. [5]).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “When the sensory inputs are high-dimensional images, latent dynamics models can abstract observations to predict forward in compact state spaces (Watter et al., 2015; Oh et al., 2017; Gregor et al., 2019).”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, methodology): “RL methods have also been able to tackle domains where perception is a challenge, due to partial observability or a complex sensory stream (Beattie et al., 2016; Gregor et al., 2019; Paine et al., 2020; Song et al., 2020; Wayne et al., 2018).”
Generative Neurosymbolic Machines (2020, methodology): “First, unlike other ConvDRAW models [14, 19, 17], StructDRAW draws not pixels but an abstract structure on feature space, i.”
What can I do here? A Theory of Affordances in Reinforcement Learning (2020, background): “Much of the work on partial models (Oh et al., 2017; Amos et al., 2018; Guo et al., 2018; Gregor et al., 2019) focuses on models that predict only some of the state variables, or that make latent-space predictions (Schrittwieser et al., 2019), rather than restricting the space of actions considered…”
Rapid Task-Solving in Novel Environments (2020, background): “Planning over belief states might be accomplished simply by storing belief states such as those developed by Gregor et al. (2019) and training on a problem distribution that requires planning over information stored in those states.”
Action and Perception as Divergence Minimization (2020, background): “Both forms have enabled learning world models for planning (Ebert et al., 2018; Ha and Schmidhuber, 2018; Zhang et al., 2019; Hafner et al., 2018; 2019a) and accelerated RL (Lange and Riedmiller, 2010; Jaderberg et al., 2016; Lee et al., 2019a; Yarats et al., 2019; Gregor et al., 2019).”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Most of the tasks we considered could potentially be learned with sufﬁcient data alone, especially if combined with more Preprint complicated techniques, for example data augmentation (Raileanu et al., 2020; Guan et al., 2021), or auxiliary generative model learning (Gregor et al., 2019).”
Using Unity to Help Solve Intelligence (2020, methodology): “• Shaping Belief States with Generative Environment Models for RL (Gregor et al., 2019): This paper utilizes three diﬀerent environments, two of which are built using our approach: – Random City : a procedurally-generated, 3D navigation environment.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Most of our tasks could potentially be learned with sufﬁcient data alone, especially if combined with more complicated techniques, for example data augmentation (Raileanu et al., 2020; Guan et al., 2021), or auxiliary generative model learning (Gregor et al., 2019).”
Auxiliary Tasks Speed Up Learning PointGoal Navigation (2020, unknown): “This means that longer horizon predictions lead to better representations, consistent with intuitions in [19].”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “…al., 2018; Ha & Schmidhuber; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang et al., 2019), learning belief representations (Gregor et al., 2019; Lee et al., 2019) and representing state similarity using the bisimulation metric (Castro, 2020; Zhang et al., 2021a; Agarwal et…”
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning (2020, methodology): “In particular we report the performance of PBL, pixel control (state-of-theart), DRAW (Gregor et al., 2019) and contrastive predictive coding (CPC) (Oord et al., 2018) in this setting.”
Deep Hierarchical Planning from Pixels (2022, background): “To remove the orthogonal challenge of learning long-term memory (Gregor et al., 2019), the history of previously activated pads is displayed at the bottom of the image.”