Curiosity-Driven Exploration by Self-Supervised Prediction

By Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell. 2017

3,014 citations recorded by the source corpus.

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward; 2) exploration with no extrinsic reward; and 3) generalization to unseen scenarios (e.g. new levels of the same game).

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Pieter Abbeel
  4. Mohit Bansal
  5. Ruslan Salakhutdinov
  6. Chelsea Finn
  7. Quoc V. Le
  8. Jianfeng Gao

In Experimental and Cognitive Psychology

  1. Sergey Levine
  2. Jeff Clune
  3. Abhinav Kumar Gupta
  4. Deepak Pathak
  5. Kristen Grauman
  6. Chelsea Finn
  7. Saurabh Gupta
  8. Caiming Xiong

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shimon Whiteson
  6. Shixiang Shane Gu
  7. Yuxi Li
  8. Abhishek Gupta

Citations in the corpus