Large-Scale Study of Curiosity-Driven Learning

By Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, Alexei A. Efros. 2018

776 citations recorded by the source corpus.

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at this https URL

Subfields: Artificial Intelligence, Computer Science Applications, Experimental and Cognitive Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Experimental and Cognitive Psychology

In Reinforcement Learning

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Inspired by the success of intrinsic motivation methods (in particular, curiosity (Burda et al., 2018; Achiam & Sastry, 2017; Pathak et al., 2017; Azar et al., 2019)), we leverage the readily available action reconstruction errors from the generative decoder as intrinsic reward signals to boost expl”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “In [143] the agent learns a next state predictor model from its experience, and uses the error of the prediction as an intrinsic reward.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Practical considerations for curiosity have been listed in recent work [63], such as using Proximal Policy Optimization (PPO) for policy optimisation.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Exploration by Random Network Distillation (2018, methodology): “It is also argued in (Burda et al., 2018) that using episodic intrinsic rewards can leak information about the task to the agent.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[48] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A.”
Continual Learning for Robotics (2019, background): “Curiosity (Burda et al., 2019) and self-supervision (Doersch and Zisserman, 2017) allow to search for new experiences (or data) and build a base of knowledge useful to achieve actual or future tasks via transfer learning (Parisi et al.”
Competitive Experience Replay (2019, background): “Compared to ICM, CER also tends to improve the speed of learning.”
Zero-Shot Reward Specification via Grounded Natural Language (2022, methodology): “We discretize the model to output 1 if it determines the current image is positive and 0 otherwise. c) Ours-base: Our base zero-shot reward model that uses the dot product between the goal text feature and image observation feature of CLIP as reward. d) Curiosity-RL (Pathak et al., 2017; Burda et al”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “[6] present, language is both compact and sufficient.”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, methodology): “In (Burda et al., 2018) the agent learns a next state predictor model from its experience, and uses the error of the prediction as an intrinsic reward.”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, methodology): “There have been many other important recent papers investigating how much agents can learn when motivated by curiosity alone [19, 128, 126, 147, 42, 56, 127, 150].”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “…slowness (Jonschkowski & Brock, 2015) (slowness), features extracted from a predictive model (Oh et al., 2015) (predictive model), features extracted from inverse models (Agrawal et al., 2016; Burda et al., 2018), and a na¨ıve baseline that uses the full state space as the representation (state).”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “Recent work formulates intrinsic motivation based on curiosity [43, 9, 27], novelty [51, 7], and empowerment [37] to improve video game playing agents (e.g., VizDoom, Super Mario) or increase object attention [27].”
Influence-Based Multi-Agent Exploration (2019, background): “, 2015) or expert-designed rewards (Wu & Tian, 2016; OpenAI, 2018), but they tend to struggle in many real-world scenarios with sparse rewards (Burda et al., 2019).”
Malthusian Reinforcement Learning (2018, background): “, for individual curiosity [2, 6, 25, 29, 30, 33], empowerment [22], or social influence [19].”
Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “Our method is also related to curiosity-driven learning [6, 7, 34], which defines intrinsic curiosity rewards to encourage the agent to explore in an environment.”