Curiosity-Driven Exploration by Self-Supervised Prediction
By Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell. 2017
3,014 citations recorded by the source corpus.
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward; 2) exploration with no extrinsic reward; and 3) generalization to unseen scenarios (e.g. new levels of the same game).
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Chelsea Finn
- Quoc V. Le
- Jianfeng Gao
In Experimental and Cognitive Psychology
- Sergey Levine
- Jeff Clune
- Abhinav Kumar Gupta
- Deepak Pathak
- Kristen Grauman
- Chelsea Finn
- Saurabh Gupta
- Caiming Xiong
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- An efficient framework for learning sentence representations (2018, background): “Loss functions defined in a feature space as opposed to the raw data space have been found to be more attractive in recent work for similar reasons (Larsen et al., 2015; Pathak et al., 2017).”
- Reward learning from human preferences and demonstrations in Atari (2018, background): “…motivation measure may include mutual information between actions and end states (Gregor et al., 2016), state prediction error or surprise (Pathak et al., 2017), state visit counts (Storck et al., 1995; Bellemare et al., 2016), distinguishability to a decoder (Eysenbach et al., 2018), or…”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “A popular class of on-policy algorithms is policy gradient methods (Williams, 1992; Sutton et al., 2000), which have been shown to be effective for a diverse array of complex tasks (Heess et al., 2017; Pathak et al., 2017; Peng et al., 2018; Rajeswaran et al., 2018).”
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Ideas from query (re)formulation [22, 59, 37, 51] may help agents expand the range of search exploration, and improved action exploration [40, 12, 49] and memory [53, 13, 23] mechanisms could help agents make better decisions over the long horizon and large action space.”
- Recurrent World Models Facilitate Policy Evolution (2018, background): “An exciting research direction is to look at ways to incorporate artificial curiosity and intrinsic motivation [61, 64, 77, 80, 81] and information seeking [23, 86] abilities in an agent to encourage novel exploration [47].”
- TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “However, this objective causes significant model capacity to be devoted to predicting irrelevant aspects of the environment dynamics, such as noisy backgrounds, at the expense of value-critical features that may occupy only a small part of the observation space (Pathak et al., 2017).”
- Continual Lifelong Learning with Neural Networks: A Review (2018, background): “ity. Recent work in (deep) reinforcement learning has included mechanisms of curiosity and intrinsic motivation to address scenarios where the rewards are sparse or deceptive (Forestier et al., 2017; Pathak et al., 2017; Tanneberg et al., 2017; Bellemare et al., 2016; Kulkarni et al., 2016). In a sc”
- Weight Agnostic Neural Networks (2019, background): “…single WANN capable of encoding many different useful tasks in its environment, one might consider developing a WANN with a strong intrinsic bias for intrinsic motivation [82, 84, 94], and continuously optimize its architecture to perform well at pursuing novelty in an open-ended environment [60].”
- Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Inspired by the success of intrinsic motivation methods—in particular, curiosity [14, 2, 75, 4]—we leverage the readily available action reconstruction errors from the generative decoder as intrinsic reward signals to boost exploration when training πg .”
- Causally Correct Partial Models for Reinforcement Learning (2020, background): “…generation of synthetic training data (Weber et al., 2017), better credit assignment (Heess et al., 2015), learning useful internal representations and belief states (Gregor et al., 2019; Guo et al., 2018), or exploration via quantification of uncertainty or information gain (Pathak et al., 2017).”
- Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Our learned action space is agnostic to the control algorithm and hence other exploration algorithms [24, 4, 26] can also benefit from using it.”
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “Others have shown benefits of intrinsic motivation for learning RL tasks (Eysenbach et al., 2018; Kulkarni et al., 2016; Pathak et al., 2017; Song et al., 2019), though the connections to policy robustness and object-based representations are not explored in these works.”
- World Models (2018, background): “An exciting research direction is to look at ways to incorporate artificial curiosity and intrinsic motivation (Schmidhu-ber, 2010; 2006; 1991b; Pathak et al., 2017; Oudeyer et al., 2007) and information seeking (Schmidhuber et al., 1994; Gottlieb et al., 2013) abilities in an agent to encourage…”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Authors of [142] define curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model.”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Noise in Parameter Space for Exploration (Fortunato et al., 2018; Plappert et al., 2018): Pertubates network parameters to aid exploration.”
- A Theoretical Study on Solving Continual Learning (2022, background): “Further afield, this work is also related to curiosity-driven self-supervised learning [77] in reinforcement learning and 3D navigation.”
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Using the same DD-PPO baseline architecture from the previous work, this work adds three auxiliary tasks: action-conditional contrastive predictive coding (CPC—A) [96], inverse dynamics [46] and temporal distance estimation.”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
- Exploration by Random Network Distillation (2018, methodology): “Another class of exploration methods rely on errors in predicting dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018).”