Uncertainty-Aware Reinforcement Learning for Collision Avoidance

By Gregory Kahn, Adam Villaflor, Vitchyr H. Pong, Pieter Abbeel, Sergey Levine. 2017

337 citations recorded by the source corpus.

Reinforcement learning can enable complex, adaptive behavior to be learned automatically for autonomous robotic platforms. However, practical deployment of reinforcement learning methods must contend with the fact that the training process itself can be unsafe for the robot. In this paper, we consider the specific case of a mobile robot learning to navigate an a priori unknown environment while avoiding collisions. In order to learn collision avoidance, the robot must experience collisions at training time. However, high-speed collisions, even at training time, could damage the robot. A successful learning method must therefore proceed cautiously, experiencing only low-speed collisions until it gains confidence. To this end, we present an uncertainty-aware model-based learning algorithm that estimates the probability of collision together with a statistical estimate of uncertainty. By formulating an uncertainty-dependent cost function, we show that the algorithm naturally chooses to proceed cautiously in unfamiliar environments, and increases the velocity of the robot in settings where it has high confidence. Our predictive model is based on bootstrapped neural networks using dropout, allowing it to process raw sensory inputs from high-bandwidth sensors such as cameras. Our experimental evaluation demonstrates that our method effectively minimizes dangerous collisions at training time in an obstacle avoidance task for a simulated and real-world quadrotor, and a real-world RC car. Videos of the experiments can be found at this https URL

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “[31, 42]); for example, Bayesian uncertainty estimates have been proposed as an alternative to double DQN [3].”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “[4] propose a collision avoidance system trained with reinforcement learning which uses uncertainty predictions to minimize the risk of collision.”
MagNet: A Two-Pronged Defense against Adversarial Examples (2017, background): “They are used in autonomous control for robots and vehicles [1, 3, 4], financial systems [32], medical treatments [31], information security [12, 29], and human-computer interaction [11, 13].”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, methodology): “1We note that [24] uses a purely prediction-based method, but focuses on uncertainty and safety.”
Concrete Dropout (2017, methodology): “Existing RL research using dropout uncertainty would hold the dropout probability fixed, or decrease it following a schedule [9, 10, 18].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “A number of recent works have studied self-supervised robotic learning, where large-scale unattended data collection is used to learn individual skills such as grasping [22], [23], [24], [25], push-grasp synergies [26], or obstacle avoidance [27], [28].”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “Our approach also uses bootstrap for uncertainty estimation, but unlike our method, Kahn et al. (2017) does not learn a reset or safety policy.”
Self-Supervised Visual Planning with Temporal Skip Connections (2017, background): “Several prior works have focused on autonomous data collection for individual skills, such as grasping [7, 8] or obstacle avoidance [9, 10].”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Because current RL algorithms can take unsafe actions [56, 57] and require tremendous amounts of experience to learn [41, 42, 58], the majority of applications of RL in the foreseeable future will likely require training in a simulator before being transferred to (and optionally fine-tuned in) the r”
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning (2018, background): “Many self-supervised robot learning methods have focused on predicting the outcome of a particular event, such as grasp success [1, 2, 3] or crashing [8, 9].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “a [173] used both Bootstrapping and Dropout methods to estimate uncertainty in NNs and then used in UA collision prediction model.”
Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation (2019, background): “Some recent approaches have demonstrated empirical success learning when to intervene, either in the form of resetting, collecting expert feedback, or falling back to a safe policy [8, 16, 19, 14].”
Deep reinforcement learning based mobile robot navigation: A review (2021, methodology): “Kahn et al. [32] focused on this problem and proposed a learning algorithm based on an uncertainty model that uses a neural network to estimate the probability of collision.”
A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks (2019, background): “Bayesian approaches to uncertainty estimation have indeed been proposed to assess the reliability of clinical predictions [24, 25, 47, 30] but have only been applied to a handful of large-scale real-world problems [30, 15, 42] that neural networks (NNs) have proven themselves particularly effective ”
Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning (2017, background): “robot motion planning, they have mainly focused on a discrete action space [13], [21] or on small-scale problems [8], [9], [16], [17].”
Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios (2020, methodology): “To overcome this limitation, Tai et al. (2017) proposed a map-less motion planner trained through a DRL method and Kahn et al. (2017) presented an uncertainty-aware model-based reinforcement-learning algorithm to estimate the probability of collision in an unknown environment.”
Mutual Alignment Transfer Learning (2017, methodology): “As MATL employs partially trained policies on the real platform, further development will have to address methods for safe application of RL on real world platforms [31, 32].”
A survey of uncertainty in deep neural networks (2021, background): “Also for fields where uncertainties form a crucial part of the learning techniques, such as for active learning [23], [24], [25], [26] or reinforcement learning [20], [27], [28], [29], uncertainty estimates are highly important.”