Recurrent World Models Facilitate Policy Evolution
By David R Ha, J. Urgen Schmidhuber. 2018
1,336 citations recorded by the source corpus.
A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatio-temporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of this paper is available at https://worldmodels.github.io
Subfields: Development, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Development
- Jimmy Ba
- Amir-massoud Farahmand
- David R Ha
- Adam Gaier
- Tingwu Wang
- Sergey Levine
- Denis Yarats
- Brandon Amos
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Jimmy Ba
- Timothy P. Lillicrap
- Yuxi Li
Citations in the corpus
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Rich image representations are useful for many downstream tasks; for example, VAE are often used in model-based reinforcement learning [14, 17, 18] for predicting future outcomes of the environment.”
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Another avenue of future work is to learn “textual dynamics models” through environment interactions, akin to vision-based world models (Ha and Schmidhuber, 2018).”
- Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “[13] David Ha and Jürgen Schmidhuber.”
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “This framework is similar to (Ha & Schmidhuber, 2018), who learn a latent space representation of the environment with an auto-encoder, and use it to train an RL agent.”
- Model-Based Reinforcement Learning for Atari (2019, background): “2017, Chapter 8), capturing important details of the scene (Ha & Schmidhuber, 2018), encouraging exploration (Oh et al., 2015), creating intrinsic motivation (Schmidhuber, 2010) or counterfactual reasoning (Buesing et al., 2019).”
- Weight Agnostic Neural Networks (2019, methodology): “We compare the mean performance (over 100 trials) of the best weight agnostic network architectures found with standard feed forward network policies commonly used in previous work (i.e. [37, 38]).”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “World models proposed in [130] are trained quickly in an unsupervised way, via a variational autoencoder (VAE), to learn a compressed spatial and temporal representation of the environment.”
- Implicit Behavioral Cloning (2021, methodology): “Explicit “MSE” models are60 trained with Mean Square Error (MSE), explicit “MDN” models are Mixture Density Networks (MDN)61 [21], and implicit “EBM” models are trained with LInfoNCE and optimized with derivative-free optimization.62 Figs.”
- Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “…has been demonstrated in some of the recent works (Levine et al., 2016; Nagabandi et al., 2018; Chua et al., 2018b; Kurutach et al., 2018; Ha & Schmidhuber, 2018), learning an accurate global model that works for all state-action pairs can be exceedingly challenging, especially for…”
- Exploring Model-based Planning with Policy Networks (2019, methodology): “Latent dynamics models using VAE [19] are commonly used to solve problems with image input [12, 13, 15, 18].”
- Improving Sample Efficiency in Model-Free Reinforcement Learning from Images (2019, methodology): “These methods use a world model (Ha & Schmidhuber, 2018) approach, learning a representation space using a latent dynamics loss and pixel decoder loss to ground on the original observation space.”
- Visual Grounding of Learned Physical Models (2020, background): “People also have studied ways of reasoning about the physics and learning forward model directly from visual inputs (Finn & Levine, 2017; Babaeizadeh et al., 2018; Hafner et al., 2019; Ha & Schmidhuber, 2018; Wu et al., 2017).”
- Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, background): “Previous studies in this field [13]– [19] train autoencoding models along with latent dynamics models that generate imagined trajectories that are used for planning or policy optimization.”
- Mechanistic Interpretability for AI Safety - A Review (2024, background): “In conclusion, the evolution of LLMs from simple predictive models to entities potentially possessing complex internal world models , as suggested by the simulation hypothesis and supported by mechanistic interpretability studies, represents a significant shift in our understanding of these systems.”
- On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “We refer to the dynamics, rewards, and termination map as the world model , e.g. following Ha and Schmidhuber (2018), and consider MDPs with known and unknown world models.”
- End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “RL can be effectively combined with world models [82], [83], [84], though this presents specific challenges (See Section IV-C).”
- Reasoning with Language Model is Planning with World Model (2023, methodology): “In general, a world model predicts the next state of the reasoning after applying an action to the current state (Ha and Schmidhuber, 2018b; Matsuo et al., 2022).”
- Deep Learning for Classical Japanese Literature (2018, methodology): “The architecture for the VAE is identical to [9] and both datasets are compressed into their own respective 64-dimensional latent space, zold and znew.”
- Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, background): “…in the problem – both in terms of the agent’s model of the world, its knowledge of its current state, or uncertainty in the mapping between actions and future rewards (Ha & Schmidhuber, 2018; Millidge, Tschantz, & Buckley, 2020; Pathak, Agrawal, Efros, & Darrell, 2017; Tschantz et al., 2020).”