EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

By Justin Fu, John D. Co-Reyes, Sergey Levine. 2017

164 citations recorded by the source corpus.

Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with count-based exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, background): “proposed, such as PAC guaranteed exploration methods targeting at continuous spaces [150], [151], concurrent exploration mechanisms [318], [319], [320] and exploration in deep RL [321], [322], [323].”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “The hyperparameters and accuracy for the TRPO and VIME results follow from the concurrent work (Fu et al., 2017).”
Exploration by Random Network Distillation (2018, background): “…are a natural and effective way to do exploration (Strehl & Littman, 2008) and a lot of work has studied how to tractably generalize count bonuses to large state spaces (Bellemare et al., 2016; Fu et al., 2017; Ostrovski et al., 2017; Tang et al., 2017; Machado et al., 2018; Fox et al., 2018).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Examples of such works include empowerment [37, 47], count-based exploration [8, 50, 67, 21], information gain about agent’s dynamics [64] and forward-inverse dynamics models [53].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Smoothed versions of state visitation counts can be used for intrinsic rewards [3, 9, 24, 47].”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “This can be thought of as a means of density based exploration, related to (Bellemare et al., 2016; Fu et al., 2017) level behavior which is trained on a “proxy” reward function.”
Learning Awareness Models (2018, methodology): “A sea of other methods have been proposed for exploration (MacKay, 1992; Ghavamzadeh et al., 2015; Asmuth et al., 2009; Gal, 2016; Stachniss et al., 2005; Plappert et al., 2017; Fu et al., 2017).”
DeepMimic (2018, methodology): “A number of algorithmic improvements have been proposed to improve exploration, such as using metrics for novelty or information gain [Bellemare et al. 2016; Fu et al. 2017; Houthooft et al. 2016].”
Episodic Curiosity through Reachability (2018, methodology): “Multiple works (Fu et al., 2017; Savinov et al., 2018; Aytar et al., 2018) train a binary classifier for predicting if the distance in time between frames is within a certain threshold or not.”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, background): “A large body of prior work has studied exploration and intrinsic motivation objectives [51, 48, 43, 22, 8, 5, 35, 42].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “They leverage the data either by making some task-agnostic assumption about the data distribution being simpler than the raw pixels (unsupervised approaches like the autoen-coder family [34, 83, 41, 48] and Generative Adversarial Networks [77, 23, 57, 15]) or they exploit some known structure (so-ca”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, background): “Another family, Generative Adversarial Networks [47, 13], formulates the problem as a game where the empirical data distribution is an optimal solution.”
Competitive Experience Replay (2019, background): “…(Bellemare et al., 2016; Tang et al., 2017), VIME (Houthooft et al., 2016), bootstrap DQN (Osband et al., 2016), goal exploration process (Forestier et al., 2017), parameter space noise (Plappert et al., 2017), dynamics curiosity and boredom (Schmidhuber, 1991), and EX2 (Fu et al., 2017).”
Meta-learning curiosity algorithms (2020, methodology): “We can also generate meaningful algorithms similar to novelty search (Lehman & Stanley, 2008) and EX(2) (Fu et al., 2017); which include buffers and nearest neighbours.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “More sophisticated exploration strategies might further alleviate these issues (Fu et al. 2017; Pathak et al. 2017), and parameter-space exploration strategies might offer a particularly promising approach to combating this issue (Burda et al. 2019).”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “[70] state encoder TRPO Memory Mujoco: SparseHalfCheetah 173.”