Active Policy Learning for Robot Planning and Exploration under Uncertainty

By Ruben Martinez-Cantin, Nando de Freitas, Arnaud Doucet, José A. Castellanos. 2007

181 citations recorded by the source corpus.

This paper proposes a simulation-based active policy learning algorithm for finite-horizon, partially-observed sequential decision processes. The algorithm is tested in the domain of robot navigation and exploration under uncertainty. In such a setting, the expected cost, that must be minimized, is a function of the belief state (filtering distribution). This filtering distribution is in turn nonlinear and subject to discontinuities, which arise because constraints in the robot motion and control models. As a result, the expected cost is non-differentiable and very expensive to simulate. The new algorithm overcomes the first difficulty and reduces the number of required simulations as follows. First, it assumes that we have carried out previous simulations which returned values of the expected cost for different corresponding policy parameters. Second, it fits a Gaussian process (GP) regression model to these values, so as to approximate the expected cost as a function of the policy parameters. Third, it uses the GP predicted mean and variance to construct a statistical measure that determines which policy parameters should be used in the next simulation. The process is then repeated using the new parameters and the newly gathered expected cost observation. Since the objective is to find the policy parameters that minimize the expected cost, this iterative active learning approach effectively trades-off between exploration (in regions where the GP variance is large) and exploitation (where the GP mean is low). In our experiments, a robot uses the proposed algorithm to plan an optimal path for accomplishing a series of tasks, while maximizing the information about its pose and map estimates. These estimates are obtained with a standard filter for simultaneous localization and mapping. Upon gathering new observations, the robot updates the state estimates and is able to replan a new path in the spirit of open-loop feedback control.

Subfields: Computer Networks and Communications, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Ocean Engineering

Citations in the corpus

Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, positive): “This interactive Bayesian optimization strategy is particulary effective as humans can be very good at comparing examples, but unable to produce an objective function whose optimum is the example of interest.”
Learning Where to Attend with Deep Architectures for Image Tracking (2011, background): “Martinez-Cantin, de Freitas, Doucet, and Castellanos ( 2007 ) applied Bayesian optimization to guide policy search so as to minimize the uncertainty in the location of visual features in the setting of a robot exploring and mapping its environment.”
Multi-fidelity Bayesian Optimisation with Continuous Approximations (2017, background): “, 2012), optimal policy search (Lizotte et al., 2007; Martinez-Cantin et al., 2007) and scientific experiments (Gonzalez et al.”
Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations (2016, background): “Some applications are hyper-parameter tuning in expensive machine learning algorithms, optimal policy search in complex systems, and scientiﬁc experiments [20, 23, 27].”
Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters (2014, background): “Bayesian optimisation has become an important area of research and development in the field of machine learning, as evidenced by recent media coverage [8] and a blossoming range of applications to interactive user-interfaces [4], robotics [16, 19], environmental monitoring [18], information extracti”
Trajectory Optimization using Reinforcement Learning for Map Exploration (2008, methodology): “The closest algorithm in spirit to our reinforcement learning algorithm uses Gaussian processes to model the predicted reward of an exploring robot ( Martinez-Cantin et al. 2007 ).”
On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning (2014, background): “…such as MCMC inference [Mahendran et al., 2012, Hamze et al., 2013, Wang et al., 2013a], deep learning [Bergstra et al., 2011], preference learning [Brochu et al., 2007, 2010a], reinforcement learning and control [Martinez-Cantin et al., 2007, Lizotte et al., 2012], and more [Snoek et al., 2012].”
A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot (2009, methodology): “Although we actually learn the mean function in the manner proposed in Martinez-Cantin et al. (2007a) , for presentation clarity let us assume that it is the zero function as it is often the case in the machine learning literature.”
High Dimensional Bayesian Optimisation and Bandits via Additive Models (2015, methodology): “, 2012), robotics (Lizotte et al., 2007; Martinez-Cantin et al., 2007) and object tracking (Denil et al.”
Gaussian process dynamic programming (2009, background): “A similar perspective to this kind of local function approximation is mentioned by Martinez-Cantin et al. (2007).”
Variational Bayesian Optimization for Runtime Risk-Sensitive Control (2012, background): “Recently there has been increased interest in applying Bayesian optimization (BO) algorithms to solve model-free policy search problems [17, 19, 20, 33, 28, 13].”
Body schema acquisition through active learning (2010, unknown): “2) Choose θ ∗ = arg min θ C ( θ ) .”
Multi-Fidelity Black-Box Optimization with Hierarchical Partitions (2018, background): “It finds applications in many real world scientific and engineering tasks including scientific experimentation, industrial design, and model selection in statistics and machine learning (Martinez-Cantin et al., 2007; Parkinson et al., 2006; Snoek et al., 2012).”
Bayesian Optimization in a Billion Dimensions via Random Embeddings (2013, methodology): “In recent years, the machine learning community has increasingly used Bayesian optimization (Rasmussen, 2003; Brochu et al., 2007; Martinez-Cantin et al., 2007; Lizotte et al., 2007; Frazier et al., 2009; Azimi et al., 2010; Hamze et al., 2011; Azimi et al., 2011; Bergstra et al., 2011; Gramacy &…”
Asynchronous Parallel Bayesian Optimisation via Thompson Sampling (2017, background): “Bayesian optimisation (BO) refers to a suite of methods for black-box optimisation under Bayesian assumptions on f that has been successfully applied in many of the above applications [11, 15, 26, 28, 35].”
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks (2017, background): “Active visual localization and mapping: Active visual SLAM aims to limit samples needed to densely reconstruct a 3D environment using geometric methods [13, 34, 35, 44, 60].”
An Entropy Search Portfolio for Bayesian Optimization (2014, methodology): “Major applications of these techniques include interactive user-interfaces [6, 4], robotics [18, 21], environmental monitoring [20], information extraction [33], combinatorial optimization [16, 34], sensor networks [10, 28], adaptive Monte Carlo [19, 32], experimental design [1], and reinforcement…”
Parallelised Bayesian Optimisation via Thompson Sampling (2018, background): “Bayesian optimisation (BO) refers to a suite of methods for blackbox optimisation under Bayesian assumptions on f that has been successfully applied in hyper-parameter tuning, drug discovery and other applications in policy search, online advertising, and scientific experimentation [12, 17, 30, 32, ”