DropoutDAgger: A Bayesian Approach to Safe Imitation Learning

By Kunal Menda, Katherine Rose Driggs-Campbell, Mykel J. Kochenderfer. 2017

32 citations recorded by the source corpus.

While imitation learning is becoming common practice in robotics, this approach often suffers from data mismatch and compounding errors. DAgger is an iterative algorithm that addresses these issues by continually aggregating training data from both the expert and novice policies, but does not consider the impact of safety. We present a probabilistic extension to DAgger, which uses the distribution over actions provided by the novice policy, for a given observation. Our method, which we call DropoutDAgger, uses dropout to train the novice as a Bayesian neural network that provides insight to its confidence. Using the distribution over the novice's actions, we estimate a probabilistic measure of safety with respect to the expert action, tuned to balance exploration and exploitation. The utility of this approach is evaluated on the MuJoCo HalfCheetah and in a simple driving experiment, demonstrating improved performance and safety compared to other DAgger variants and classic imitation learning.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “While our target setting is ofﬂine data collection, online imitation learning is an active area of research in the case of hybrid IL-RL (Ross & Bagnell, 2014; Sun et al., 2018) and “safe” IL (Sun et al., 2017; Menda et al., 2017; Zhang & Cho, 2017).”
On the Sample Complexity of Stability Constrained Imitation Learning (2021, methodology): “[11] Kunal Menda, Katherine Driggs-Campbell, and Mykel J.”
Safe end-to-end imitation learning for model predictive control (2018, methodology): “The work most similar to our own is the DropoutDAgger approach [7], a probabilistic extension to DAgger which uses the Dropout method of Bayesian approximation first presented in [8] to obtain a predictive distribution from a neural network.”
Early Failure Detection of Deep End-to-End Control Policy by Reinforcement Learning (2019, methodology): “However, in [6] both the dropout probability and the uncertainty threshold are selected manually; the tasks and experts considered are based on RL; and the case of partially observable environments with vision-based tasks is not considered at all.”