Exploration by Random Network Distillation

By Yuri Burda, Harrison Edwards, Amos J. Storkey, Oleg Klimov. 2018

1,706 citations recorded by the source corpus.

We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Alternate prior approaches to offline RL estimate some sort of uncertainty to determine the trustworthiness of a Q-value prediction [30, 3, 34], typically using uncertainty estimation techniques from exploration in online RL [47, 26, 46, 7].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Random Distillation Network [82] is one method that predicts the output of a randomly initialized neural network, as the answer is a deterministic function of its inputs.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “In order to fairly compare all prior methods, we use the same underlying RL algorithm, soft actor-critic (SAC) [22], across all methods, and use the same exploration bonus used in R3L [54], random network distillation [7], which is added to the reward for all methods.”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Unknown state-action detector (USAD): In order to partition the state-action space into known and unknown regions, we use uncertainty quantification [69, 70, 71, 72].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This includes demonstration data, data from modern exploration methods that scale to pixel observations [51, 52, 53], and task-driven data from running reinforcement learning on particular tasks.”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “We recall that the computation of the policy improvement step relies on the estimates of an error function eP , which may, for instance, be indirectly inferred from pseudo-counts ÑD(x, a) (Bellemare et al., 2016; Fox et al., 2018; Burda et al., 2019).”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “We recall that the computation of the policy improvement step relies on the estimates of an error function eP , which may, for instance, be indirectly inferred from pseudo-counts ÑD(x, a) (Bellemare et al., 2016; Fox et al., 2018; Burda et al., 2019).”
Hyperparameter Selection for Imitation Learning (2021, methodology): “As proposed by Burda et al. (2018); Wang et al. (2019), we define a support estimation metric by taking two randomly initialized networks, and train one to predict the random output of the other one on the expert training set.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “0.0 (0.0) 0.6 (0.6) 2.3 (0.3) 1.6 (0.1) 2.0 (0.2) 0.0 (0.0) #Hunt 0.7 (0.1) 0.6 (0.3) 0.3 (0.0) 0.7 (0.0) 0.9 (0.1) 0.7 (0.0) (Burda et al., 2019).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, methodology): “Todor created the manipulation tasks in the transfer suite, helped Yi with the RND baseline, and prepared code for open-sourcing.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “However, in principle D ( s, a ) can be computed with a number of pseudo-count methods [Bellemare et al., 2016, Tang et al., 2017, Burda et al., 2018].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “One solved level 1 of the game in 10% of its runs [16].”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, methodology): “Since our approach does not require any reward information and is agnostic as to the underlying RL algorithm, it would naturally complement exploration methods designed for sparse reward problems (Pathak et al., 2017; Burda et al., 2018).”
Influence-Based Multi-Agent Exploration (2019, background): “These works result in significant progress in many challenging tasks such as Montezuma Revenge (Burda et al., 2018), robotic manipulation (Pathak et al.”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “The literature on novelty-driven exploration describes several approaches to deriving state representations [7].”