Model based Bayesian Exploration

By Richard Dearden, Nir Friedman, David Andre. 1999

Reinforcement learning systems are often concerned with balancing exploration of untested actions against exploitation of actions that are known to be good. The benefit of exploration can be estimated using the classical notion of Value of Information - the expected improvement in future decision quality arising from the information acquired by exploration. Estimating this quantity requires an assessment of the agent's uncertainty about its current value estimates for states. In this paper we investigate ways to represent and reason about this uncertainty in algorithms where the system attempts to learn a model of its environment. We explicitly represent uncertainty about the parameters of the model and build probability distributions over Q-values based on these. These distributions are used to compute a myopic approximation to the value of information for each action and hence to select the action that best balances exploration and exploitation

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “Another model-based Bayesian approach was proposed by Dearden, Friedman, and Andre [18], where the Value of Information for exploring states is computed considering a model and uncertainty about its parameters.”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, background): “In the reinforcement learning literature, Bayesian approaches have also been explored for similar reasons ( Dearden et al. 1999 ; S trens2000 ;W ang et al.2005; Engel et al. 2005); other methods targeting efficient exploration have also been explored (Kakade et al. 2003; Strehl et al. 2006; Brunskil”
Benchmarking Model-Based Reinforcement Learning (2019, background): “While exploration and off-policy learning have been studied in [2, 10, 51, 23, 42, 18], it has been barely addressed on current model-based approaches.”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Computational methods exist for both parametric [Dearden et al., 1999, Poupart and Vlassis, 2008] and non-parametric [Doshi-Velez et al., 2012, 2015] learning of the reward function when the IRL problem itself is modeled as a POMDP.”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “Previously, Bayesian RL methods have been explored to address these drawbacks [12, 13].”
Evolved Policy Gradients (2018, methodology): “In (4), n algorithm named MAML is introduced.”
An empirical evaluation of interval estimation for Markov decision processes (2004, background): “Many approaches to exploration in MDPs have been proposed (Thrun, 1992; Wiering & Schmidhuber, 1998; Dearden et al., 1999; Meuleau & Bourgine, 1999; Wyatt, 2001).”
Bayesian role discovery for multi-agent reinforcement learning (2010, unknown): “Previous work on Bayesian RL has considered priors on the domain dynamics in model-based RL [1, 7] and priors on expected returns, and action-value functions [2, 3].”
Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning (2017, background): “Within reinforcement learning, Bayesian notions of model uncertainty have a long history (Schmidhuber, 1991b;a; Dearden et al., 1999; Still & Precup, 2012; Sun et al., 2011; Maddison et al., 2017).”
Robust planning with incomplete domain models (2017, methodology): “Reinforcement learning, and in particular the variant called bayesian reinforcement learning (Dearden et al., 1999; Strens, 2000) aims to learn a posterior over the domain models, and plan (determine behavior) using this distribution over the models.”
Coordination in multiagent reinforcement learning: a Bayesian approach (2003, methodology): “Various techniques 3.2 Computational Approximations for making this process more ef.cient can be used as well, includ-Solving the belief state MDP above will generally be computa-ing importance sampling (allowing results from one MDP to be tionally infeasible.”
Learning and Using Models (2012, methodology): “Model Based Bayesian Exploration (MBBE) is a similar approach taken by Dearden et al (1999).”
Multi-task reinforcement learning: a hierarchical Bayesian approach (2007, background): “Hierarchical Bayesian RL is also related to Bayesian Reinforcement Learning ( Dearden et al., 1998a; Dearden et al., 1998b; Strens, 2000; Du, 2003), where the goal is to give a principled solution to the problem of exploration by explicitly modeling the uncertainty in the rewards, state-transition ”
Offline Policy Selection under Uncertainty (2020, background): “…many Bayesian-inspired methods have been proposed for policy optimization (Deisenroth and Rasmussen, 2011; Parmas et al., 2018), especially in the context of exploration (Houthooft et al., 2016; Dearden et al., 2013; Kolter and Ng, 2009), relatively few have been proposed for policy evaluation.”
Learning Montezuma's Revenge from a Single Demonstration (2018, methodology): “Prior works (Dearden et al. [1999], Kolter and Ng [2009], Tang et al. [2017], Ostrovski et al. [2017], Chen et al. [2017], Pathak et al. [2017], O’Donoghue et al. [2016], Schulman et al. [2017a], Haarnoja et al. [2017], Nachum et al. [2017]) have proposed various methods of overcoming the…”
Thompson Sampling for Learning Parameterized Markov Decision Processes (2014, background): “In the canonical parameterization, the parameter θ factors into separate components for each state and action (Dearden et al., 1999).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “When rewards are not sparse, a measure of the uncertainty on the value function can be used to drive the exploration (Dearden et al., 1998; Dearden et al., 1999).”
Percentile Optimization for Markov Decision Processes with Parameter Uncertainty (2010, background): “This example illustrates the need for criteria that address parameter uncertainty in general and specifically in the MDPs (e.g., Ben-Tal and Nemirovski 1998, Silver 1963, Martin 1967, Satia and Lave 1973, Dearden et al. 1999).”