Risk-Aware Active Inverse Reinforcement Learning

By Daniel S. Brown, Yuchen Cui, Scott Niekum. 2018

63 citations recorded by the source corpus.

Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Recent work on active learning for inverse RL has sought to reduce the required number of demonstrations [2, 6, 25, 5], but still requires some number of demonstrations to be provided manually.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “hly unsure of what to do. Some approaches actively ask users for additional (partial) demonstrations in areas of the state space in which condence is low (Chernova and Veloso, 2009) or risk is high (Brown et al., 2018), while others rely on a human user to identify when a mistake has been made (Niek”
Learning to Interactively Learn and Assist (2019, background): “…training; either continuously (Ross, Gordon, and Bagnell 2011b; Kelly et al. 2018) or at the agent’s discretion (Ross, Gordon, and Bagnell 2011a; Borsa et al. 2017; Xu et al. 2018; Hester et al. 2018; James, Bloesch, and Davi-son 2018; Yu et al. 2018a; Krening 2018; Brown, Cui, and Niekum 2018).”
Bayesian Robust Optimization for Imitation Learning (2020, methodology): “VaR-BIRL [11, 12], a closely-related method, uses VaR, a risk measure, to quantify the robustness of a given policy in Bayesian IRL.”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, methodology): “Prior work on high-conﬁdence policy evaluation for imitation learning has used Bayesian inverse reinforcement learning (IRL) Ramachandran & Amir (2007) to allow an agent to reason about reward uncertainty and policy generalization error (Brown et al., 2018).”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, methodology): “Our ﬁrst application is to provide a lower bound on the sample complexity of learning a reward function via active queries (Lopes, Melo, and Montesano 2009; Cui and Niekum 2017; Brown, Cui, and Niekum 2018).”
Learning Human Objectives by Evaluating Hypothetical Behavior (2019, background): “In step (2) we represent ˆ R by classifying state transitions as good, unsafe, or neutral – similar to Cui & Niekum (2018) – and assigning a known, constant reward to each of these three categories: ˆ R ( s, a, s (cid:48) ) = c ∈{ good , unsafe , neutral } p θ ( c | s, a, s (cid:48) ) R c , (2)…”