World Models

By David R Ha, J. Urgen Schmidhuber. 2018

1,897 citations recorded by the source corpus.

We explore building generative neural network models of popular reinforcement learning environments. Our world model can be trained quickly in an unsupervised manner to learn a compressed spatial and temporal representation of the environment. By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment. An interactive version of this paper is available at https://worldmodels.github.io/

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “…to each question that they answer and (b) seem to be limited in scope, relying to 16 Interesting work towards integrating more Ha and Schmidhuber (Ha & Schmidhuber, 2018) and (Veerapaneni et al., 2019). a large degree on verbal overlap between question and answer and (c) have very limited…”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
Model-Based Reinforcement Learning for Atari (2019, background): “2017, Chapter 8), capturing important details of the scene (Ha & Schmidhuber, 2018), encouraging exploration (Oh et al., 2015), creating intrinsic motivation (Schmidhuber, 2010) or counterfactual reasoning (Buesing et al., 2019).”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “…learning by training on imagined experience (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018), improving feature representations (Wayne et al., 2018; Igl et al., 2018), or leveraging the information content of the…”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “A similar approach to GATS is concurrently developed and empirically studied on Car Racing and VizDoom (David Ha, 2018).”
Toward Causal Representation Learning (2021, background): “Particularly relevant for causal leaning are generative world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see I NTRODUCTION above) to train RL agents [48], [99], [127], [178], [214], [231], [248], [268], [271].”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Another direction in this area is to learn a world model [4, 37, 36] that generates latent states [88, 38, 77].”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “ls. The future return is estimated by rolling out games until their end and, therefore, capturing delayed rewards. Recently, also forward world models using an evolution strategy have been successful [35]. However, forward view approaches fail at MDPs with delayed rewards if rewards, state transitio”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “Within MBRL, commonly explored methods include action-conditional, next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et…”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50].”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “This result for VAE models has also been conﬁrmed by [16]. been expressed as the mutual information of a single latent dimension Z i with generative factors G 1 , . . . , G K [47], where in the ideal case each Z i has some mutual information with one generative factor G k but none with all the…”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “In this sense, a more related work to our model is probably the world model (Ha and Schmidhuber, 2018), because both models used variational inference to estimate the actual state from the observation.”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “…would be interesting to use it in a reinforcement learning setting in conjunction with Imagination-Augmented Agents (Weber et al., 2017) or more generally as a world model (Ha and Schmidhuber, 2018), especially for settings with simple backgrounds, e. g., games like Montezuma’s Revenge or Pacman.”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “Synthesizer outputs modeled with a variational autoencoder were recently used as a “world model” (Ha & Schmidhuber, 2018) to pass approximate gradients to a controller during learning (Esling et al., 2019).”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Several algorithms have been proposed for simple, synthetic images [13] and video game environments [14], [15], [16], but have not been evaluated on generalization or in the real world, while other work has also studied model-based RL for individual robotic skills [17], [18], [19].”