Model-Based Active Exploration

By Pranav Shyam, Wojciech Jaśkowski, Faustino J. Gomez. 2018

199 citations recorded by the source corpus.

Efficient exploration is an unsolved problem in Reinforcement Learning which is usually addressed by reactively rewarding the agent for fortuitously encountering novel situations. This paper introduces an efficient active exploration algorithm, Model-Based Active eXploration (MAX), which uses an ensemble of forward models to plan to observe novel events. This is carried out by optimizing agent behaviour with respect to a measure of novelty derived from the Bayesian perspective of exploration, which is estimated using the disagreement between the futures predicted by the ensemble members. We show empirically that in semi-random discrete environments where directed exploration is critical to make progress, MAX is at least an order of magnitude more efficient than strong baselines. MAX scales to high-dimensional continuous environments where it builds task-agnostic models that can be used for any downstream task.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…and Darrell, 2017; Karl, Soelch, Becker-Ehmck, Benbouzid, van der Smagt, and Bayer, 2017; Burda, Edwards, Pathak, Storkey, Darrell, and Efros, 2018; Shyam, Jaśkowski, and Gomez, 2019; Liu, Trott, Socher, and Xiong, 2019; Ecoffet, Huizinga, Lehman, Stanley, and Clune, 2019; Sekar, Rybkin,…”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “Other works also propose to use ensemble of networks for measuring uncertainty (Pathak et al., 2019; Shyam et al., 2019).”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, background): “, 2017), divergence between ensembles (Chua et al., 2018; Shyam et al., 2019) and uncertain state bonuses (Bellemare et al.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Unsupervised pretraining or self-supervised learning using world models (Ha and Schmid-huber, 2018; Shyam et al., 2018; Pathak et al., 2019; Sharma et al., 2020; Sekar et al., 2020) push against the tabula rasa viewpoint of agents starting from zero knowledge about the environment it is interacting…”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, background): “Some prior works consider unsupervised pre-training for efficient RL, but these works typically focus on settings in which exploration is not as much of a challenge [63, 17, 14], focus on goal-conditioned policies [44, 40], or have not been shown to scale to high-dimensional visual observation space”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Intrinsic rewards can be derived from various measures: novelty [43, 55, 6, 56], prediction-error [38, 3], ensemble disagreement [11, 39, 48, 46, 18, 50], or information gain [23].”
World Discovery Models (2019, unknown): “…model’s prediction error, also known as prediction gain (Achiam & Sastry, 2017; Schmidhuber, 1991a; Lopes et al., 2012), and ﬁnally information gain (Shyam et al., 2018; Itti & Baldi, 2009; Little & Sommer, 2013; Frank et al., 2014; Houthooft et al., 2016). iv the last component is an RL…”
Ready Policy One: World Building Through Active Learning (2020, background): “, acquiring data in a principled manner) in MBRL have been considered previously, for example in [47, 40, 23].”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “…has typically focused on encouraging novelty-seeking behaviors by maximizing model uncertainty (Houthooft et al., 2016; Still & Precup, 2012; Shyam et al., 2018; Pathak et al., 2019), by maximizing model prediction error or improvement (Lopes et al., 2012; Pathak et al., 2017), through…”
Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control (2021, background): “Recent active exploration methods [89] and active learning of nonlinear system identification [90] suggest exploration techniques using trajectory planning to efficiently explore high uncertainty regions of the feature space.”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “A similar approach was developed in [125], using the information gain among the environmental models to decide where to go.”
Action and Perception as Divergence Minimization (2020, background): “Model parameters lead to agents that discover the rules of their environment, such as in active inference (Friston et al., 2015), VIME (Houthooft et al., 2016), MAX (Shyam et al., 2018), and Plan2Explore (Sekar et al., 2020).”
Self-Supervised Exploration via Disagreement (2019, unknown): “Concurrent to this work, Shyam et al. (2019) also show the effectiveness of model-based exploration in estimating novelty, and Henaff et al. (2019) use variance regularization for policy learning via imitation.”
Planning to Explore via Self-Supervised World Models (2020, methodology): “Plan2Explore adapts rapidly, producing effective behavior competitive to state-of-the-art supervised reinforcement learning in just a few collected episodes. the art unsupervised agents: Curiosity (Pathak et al., 2017) and Model-based Active Exploration (Shyam et al., 2019, MAX).”
On Warm-Starting Neural Network Training (2019, background): “Excluding the shrink-perturb trick, it does not appear that generally applicable techniques exist for deep neural networks that do not damage generalization, so models are typically retrained from scratch [6, 32].”
Combating the Compounding-Error Problem with a Multi-step Model (2019, methodology): “…Gal et al. , 2018], hallucinating training examples [Talvitie, 2014; Azizzadenesheli et al. , 2018; Venkatraman et al. , 2015; Oh et al. , 2015; Bengio et al. , 2015], using ensembles [Kurutach et al. , 2018; Shyam et al. , 2018], or model regularization [Jiang et al. , 2015; Asadi et al. , 2018b].”