Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty

By Nicolas Meuleau, Paul Bourgine. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DORA The Explorer: Directed Outreaching Reinforcement Action-Selection (2018, methodology): “The idea of using reinforcement-learning techniques to estimate exploration can be traced back to Storck et al. (1995) and Meuleau & Bourgine (1999) who also analyzed propagation of uncertainties and exploration values.”
An empirical evaluation of interval estimation for Markov decision processes (2004, background): “Many approaches to exploration in MDPs have been proposed (Thrun, 1992; Wiering & Schmidhuber, 1998; Dearden et al., 1999; Meuleau & Bourgine, 1999; Wyatt, 2001).”
Bayesian Reinforcement Learning (2010, methodology): “Early approximate online RL algorithms were based on conﬁdence intervals (Kael-bling, 1993; Meuleau and Bourgine, 1999; Wiering, 1999) or the value of perfect information (VPI) criterion for action selection (Dearden et al, 1999), both resulting in myopic action selection strategies.”
An analytic solution to discrete Bayesian reinforcement learning (2006, methodology): “To date, several approximate algorithms based on confidence intervals (Kaelbling, 1993; Meuleau & Bourgine, 1999), Normal-Gamma distributions (Dearden et al., 1998), linear combinations of hyperparameters (Duff, 2003) and sampling (Dearden et al., 1999; Strens, 2000; Wang et al., 2005) have been…”
Using MDP Characteristics to Guide Exploration in Reinforcement Learning (2003, background): “Interval Estimation (IE) [3, 16], as well as its global equivalent, IEQL+ [7], bias exploration toward actions that have the highest variance in the action value samples.”
The many faces of optimism: a unifying approach (2008, methodology): “Confldence interval estimation can be phrased as an exploration bonus method: see IEQL+ ( Meuleau & Bourgine, 1999 ) or MBIE-EB (Strehl & Littman, 2006).”
Optimistic Bayesian Sampling in Contextual-Bandit Problems (2012, background): “In particular, consider the methodology of evaluating both an exploitative value estimate and an 'exploratory bonus' at each time step for each action, and then acting greedily based on the sums of exploitative and exploratory values (Meuleau and Bourgine, 1999).”
Sample Complexity Bounds of Exploration (2012, methodology): “Advanced techniques exist that guide exploration using interval estimation (Kaelbling, 1993; Meuleau and Bourgine, 1999; Wiering and Schmidhuber, 1998), information gain (Dearden et al, 1999), MDP characteristics (Ratitch and Precup, 2002), or expert demonstration (Abbeel and Ng, 2005), to name a…”
Multi-armed Bandit Algorithms and Empirical Evaluation (2005, background): “There is, however, a significant amount of literature dealing with MDPs, see [17, 6] for a review.”
Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search (2012, unknown): “p, 2010] 371.5 3 59.3 4 21.8 2 671.3 126 BEB [Kolter and Ng, 2009] 386 0 67.5 3 10 1 184.6 35 Bayesian DP* [Strens, 2000] 377 1 - - - Bayes VPI+MIX* [Dearden et al., 1998] 326 31 - - 817.6 29 IEQL+* [Meuleau and Bourgine, 1999] 264 1 - - 269.4 1 QL Boltzmann* 186 1 - - 195.2 20 5 Experiments We pres”
Accelerating Reinforcement Learning through Implicit Imitation (2003, methodology): “…we would like to take not only the uncertainty of the model at the current state into account, but also the uncertainty of future states as well (Meuleau & Bourgine, 1999). nism we adopt is to require the observer to perform a (possibly augmented) Bellman backup at state s whenever the mentor…”