Planning to Explore via Self-Supervised World Models

By R. Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak. 2020

514 citations recorded by the source corpus.

Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Reasoning with Language Model is Planning with World Model (2023, methodology): “Recent years have witnessed successful applications of planning algorithms in RL [50], such as AlphaZero [52], MuZero [48].”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “, 2019), where the model-based value expansion and SVG can guide the agent away from undesirable parts of the state space; 3) unsupervised pretraining or self-supervised learning using world models (Shyam et al., 2018; Pathak et al., 2019; Sharma et al., 2020; Sekar et al., 2020) push against the ta”
Mastering Atari with Discrete World Models (2020, background): “, 2019), directed exploration (Sekar et al., 2020), and generalization from offline datasets (Yu et al.”
Deep Hierarchical Planning from Pixels (2022, methodology): “Plan2Explore still achieves some reward with four pads, struggles with ﬁve pads, and completely fails with six pads.”
DayDreamer: World Models for Physical Robot Learning (2022, background): “By predicting future outcomes, world models allow for planning and behavior learning given only small amounts of real world interaction (Gal et al., 2016; Ebert et al., 2018).”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, background): “Learning world models in either game [19–21, 43, 47] or lab environments [12, 14, 67] has been widely studied.”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “…Hafner et al., 2019; Lowrey et al., 2019; Kaiser et al., 2020; Bhard-waj et al., 2020; Yu et al., 2020; Schrittwieser et al., 2020; Nguyen et al., 2021) or for training a model-free algorithm with generated data (Pong et al., 2018; Ha & Schmidhu-ber, 2018; Hafner et al., 2020b; Sekar et al., 2020).”
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, methodology): “RSSM is used as an essential component of various MBRL methods from pixels [15]–[17], [19], [31] including Dreamer.”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Intrinsic rewards can be derived from various measures: novelty [43, 55, 6, 56], prediction-error [38, 3], ensemble disagreement [11, 39, 48, 46, 18, 50], or information gain [23].”