Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

By C. Freeman, Luke Metz, David R Ha. 2019

39 citations recorded by the source corpus.

Much of model-based reinforcement learning involves learning a model of an agent's world, and training an agent to leverage this model to perform a task more efficiently. While these models are demonstrably useful for agents, every naturally occurring model of the world of which we are aware---e.g., a brain---arose as the byproduct of competing evolutionary pressures for survival, not minimization of a supervised forward-predictive loss via gradient descent. That useful models can arise out of the messy and slow optimization process of evolution suggests that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances. Crucially, this optimization process need not explicitly be a forward-predictive loss. In this work, we introduce a modification to traditional reinforcement learning which we call observational dropout, whereby we limit the agents ability to observe the real environment at each timestep. In doing so, we can coerce an agent into learning a world model to fill in the observation gaps during reinforcement learning. We show that the emerged world model, while not explicitly trained to predict the future, can help the agent learn key skills required to perform well in its environment. Videos of our results available at https://learningtopredict.github.io/

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Rich image representations are useful for many downstream tasks; for example, VAE are often used in model-based reinforcement learning [14, 17, 18] for predicting future outcomes of the environment.”
The alignment problem from a deep learning perspective (2022, methodology): “In a simple car-racing environment, Freeman et al. [2019] found ‘emergent’ prediction models: models trained only with model-free RL that still learned to predict the outcomes of actions as a by-product.”
Ready Policy One: World Building Through Active Learning (2020, background): “We observe that stateaction pairs which cause high uncertainty/disagreement in the world model are likely to be parts of the state space that our world model is poor at modeling (as observed in (Freeman et al., 2019), where solving ‘difficult’ environment dynamics is important for learning optimal p”
Neuroevolution of self-interpretable agents (2020, background): “ign agents that can encode and process abstract concepts. Recent works have demonstrated the importance of learning abstract information from visual inputs. For example, the world models line of work [27, 38, 39, 51] learns compact representations of input image sequences, and have shown that agents”