Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks

By Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel. 2016

80 citations recorded by the source corpus.

Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “…by Bayes by Backprop trained neural networks has been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments [4, 42, 28], for training recurrent neural networks [11], but has not been applied to convolutional neural networks to-date.”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, methodology): “8 shows the target task reward as a function of training iteration for our approach alongside VIME (Houthooft et al., 2016) and SimHash (Tang et al.”
Graph networks as learnable physics engines for inference and control (2018, background): “…for learning system identiﬁcation explicitly (Yu et al., 2017; Peng et al., 2017), learning policies that are robust to hidden property variations (Rajeswaran et al., 2016), and learning exploration strategies in uncertain settings (Schmidhuber, 1991; Sun et al., 2011; Houthooft et al., 2016).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “Mutual information can been used as an objective to encourage exploration in reinforcement learning (Houthooft et al., 2016; Mohamed & Rezende, 2015).”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, background): “However, many of these approaches are not directly applicable to high-dimensional redundant continuous action spaces [12, 15], or face complexity challenges to be applicable to real world robots [16, 17].”
BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems (2016, background): “…efﬁcient exploration strategies are known for problems with ﬁnite states/actions or problems with nice structures (Kakade 2003; Asmuth et al. 2009; Jaksch, Ortner, and Auer 2010; Li et al. 2011; Osband, Russo, and Roy 2013), less is known for the general case, especially when general nonlinear…”
A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning (2019, methodology): “Recent research has also shown that adding an exploration “bonus” to the reward function can be an effective way of improving exploration; VIME [6] takes a Bayesian approach by maintaining a model of the dynamics of the environment, obtaining a posterior of the model after taking an action, and usin”
Efﬁcient Exploration for Dialogue Policy Learning with BBQ Networks & Replay Buffer Spiking (2016, background): “More recently, Stadie et al. [2015] and Houthooft et al. [2016] introduced approaches to encourage exploration by perturbing the reward function to reward the agent for discovering new dynamics of the environment.”
GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms (2017, methodology): “In deep RL, this is achieved using count-based exploration (Bellemare et al., 2016), as well as various forms of intrinsic rewards and exploration bonuses (Houthooft et al., 2016a;b; Pathak et al., 2017) and methods for option discovery (Machado et al., 2017a).”
Transfer Reinforcement Learning with Shared Dynamics (2017, methodology): “In order to guide the exploration, we adopt the well-known optimism in the face of uncertainty heuristic, which can be found in Prioritized Sweeping (Moore and Atkeson 1993), R-MAX (Brafman and Tennenholtz 2002), UCRL (Auer and Ortner 2007), and VIME (Houthooft et al. 2016).”
DeepMimic (2018, methodology): “A number of algorithmic improvements have been proposed to improve exploration, such as using metrics for novelty or information gain [Bellemare et al. 2016; Fu et al. 2017; Houthooft et al. 2016].”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Recently, the uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully to train feedforward neural networks in both supervised and reinforcement learning environments [5, 7, 8], for training recurrent neural networks [9], and for CNNs [10, 11].”
A Developmental Approach to Machine Learning? (2017, background): “Another approach uses curiosity to shift attention to new learning problems as learning progresses (Oudeyer, 2004; Houthooft et al., 2016; see also Kidd and Hayden, 2015).”
Bayesian Convolutional Neural Networks with Variational Inference (2018, unknown): “…by Bayes by Backprop trained neural networks has been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments [1, 22, 13], for training recurrent neural networks [5], but has not been applied to convolutional neural networks to-date.”
Bayesian Convolutional Neural Networks (2018, background): “…for training feedforward neural networks in both supervised and reinforcement learning environments (Blundell et al. 2015, Lipton et al. 2016, Houthooft et al. 2016), for training recurrent neural networks (Fortunato, Blundell, and Vinyals 2017), but has not been applied to convolutional…”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “For reward shaping, we compare with VIME (Houthooft et al., 2016), an exploration method based on novelty bonuses.”
Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking (2016, methodology): “More recently, Stadie et al. [2015] and Houthooft et al. [2016] introduced approaches to encourage exploration by perturbing the reward function.”