Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs

By Finale Doshi-Velez, Joelle Pineau, Nicholas A. Roy. 2008

140 citations recorded by the source corpus.

Partially Observable Markov Decision Processes (POMDPs) have succeeded in planning domains that require balancing actions that increase an agent's knowledge and actions that increase an agent's reward. Unfortunately, most POMDPs are defined with a large number of parameters which are difficult to specify only from domain knowledge. In this paper, we present an approximation approach that allows us to treat the POMDP model parameters as additional hidden state in a "model-uncertainty" POMDP. Coupled with model-directed queries, our planner actively learns good policies. We demonstrate our approach on several POMDP problems.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

An Algorithmic Perspective on Imitation Learning (2018, methodology): “Computational methods exist for both parametric [Dearden et al., 1999, Poupart and Vlassis, 2008] and non-parametric [Doshi-Velez et al., 2012, 2015] learning of the reward function when the IRL problem itself is modeled as a POMDP.”
Abstraction Levels for Robotic Imitation: Overview and Computational Approaches (2010, background): “Some authors address this problem adopting a computer vision perspective [82], modeling partial observability [40] or being robust to noise in the demonstration [26, 80, 116].”
Reinforcement Learning with Limited Reinforcement : Using Bayes Risk for Active Learning in POMDPs Finale (2012, background): “1This work expands upon work previously published by the auth ors [19]”
Planning under Uncertainty for Robotic Tasks with Mixed Observability (2010, background): “Point-based algorithms can also benefit from the ADD representation [22].”
Multi-task Reinforcement Learning in Partially Observable Stochastic Environments (2009, methodology): “consider three different Π’s, which respectively have a probability pquery = 5%, 30%, 50% of choosing the actions suggested by PBVI, where pquery corresponds to the rate of meta-query in (Doshi et al. 2008).”
Active Imitation Learning via Reduction to I.I.D. Active Learning (2012, background): “Several studies have considered active learning for reinforcement learning (RL) (Clouse 1996; Mihalkova and Mooney 2006; Gil, Stern, and Edan 2009; Doshi, Pineau, and Roy 2008), where learning is based on both autonomous exploration and queries to an expert.”
Integrating expert knowledge into POMDP optimization for spoken dialog systems (2008, background): “In the literature, expert knowledge has also been incorporated via active learning (Doshi, Pineau, and Roy 2008).”