Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

By Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua. 2016

940 citations recorded by the source corpus.

Autonomous driving is a multi-agent setting where the host vehicle must apply sophisticated negotiation skills with other road users when overtaking, giving way, merging, taking left and right turns and while pushing ahead in unstructured urban roadways. Since there are many possible scenarios, manually tackling all possible cases will likely yield a too simplistic policy. Moreover, one must balance between unexpected behavior of other drivers/pedestrians and at the same time not to be too defensive so that normal traffic flow is maintained. In this paper we apply deep reinforcement learning to the problem of forming long term driving strategies. We note that there are two major challenges that make autonomous driving different from other robotic tasks. First, is the necessity for ensuring functional safety - something that machine learning has difficulty with given that performance is optimized at the level of an expectation over many instances. Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario. We make three contributions in our work. First, we show how policy gradient iterations can be used without Markovian assumptions. Second, we decompose the problem into a composition of a Policy for Desires (which is to be learned) and trajectory planning with hard constraints (which is not learned). The goal of Desires is to enable comfort of driving, while hard constraints guarantees the safety of driving. Third, we introduce a hierarchical temporal abstraction we call an "Option Graph" with a gating mechanism that significantly reduces the effective horizon and thereby reducing the variance of the gradient estimation even further.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Constrained Policy Optimization (2017, background): “Shalev-Shwartz et al. (2016) avoid the problem of enforcing constraints on parametrized policies by decomposing ‘desires’ from trajectory planning; the neural network policy learns desires for behavior, while the trajectory planning algorithm (which is not learned) selects ﬁnal behavior and…”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Authors of [142] addressed safety in multi-agent Reinforcement Learning for Autonomous Driving, where a balance is maintained between unexpected behavior of other drivers or pedestrians and not to be too defensive, so that normal traffic flow is achieved.”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, background): “, autonomous driving [16], game playing [12] and robotics control [9]) today require reinforcement learning tasks to be carried out in multi-agent settings.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “Some investigated applications include, robotics [30], and self-driving cars [31].”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, background): “MARL has been widely applied to many real-world applications, such as autonomous driving [27], game playing [20] and swarm robotics [15].”
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving (2018, methodology): “Although visual perception tasks have been well studied by resorting to supervised learning on large-scale datasets [4,5], simplistic driving policies by manually designed rules in the modular pipeline is far from sufﬁcient for handling diverse real-world cases as discussed in [6,7].”
Transferring Autonomous Driving Knowledge on Simulated and Real Intersections (2017, positive): “Additionally, machine learning techniques have been used to optimize comfort in a space where solutions are constrained to safe trajectories (Shalev-Shwartz et al., 2016).”
Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge (2017, background): “Shalev-Shwartz et al. [14] decompose the driving strategy into a learnable part which estimate the comfort of driving, and a non-learnable part which is hard constrains on the safety of driving.”
Near-Optimal Reinforcement Learning with Self-Play (2020, background): “Multi-agent RL has achieved significant recent success on traditionally challenging tasks, for example in the game of GO [30, 31], Poker [6], real-time strategy games [33, 22], decentralized controls or multiagent robotics systems [5], autonomous driving [27], as well as complex social scenarios suc”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “RL was also used to learn the difficult double-merge scenario with multiple agents [61] each trained by a policy gradient”
Combating Deep Reinforcement Learning's Sisyphean Curse with Intrinsic Fear (2016, background): “, 2016), energy management (Night, 2016), and self-driving cars (Shalev-Shwartz et al., 2016).”
Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent (2019, background): “As there are many possible scenarios, manually tackling all possible cases using supervised learning methods will likely yield a more simplistic policy (Shalev-Shwartz et al., 2016).”
QOPT: Optimistic Value Function Decentralization for Cooperative Multi-Agent Reinforcement Learning (2020, background): “The progress of multi-agent reinforcement learning (MARL) has been arguably slow despite its importance in many applications such as controlling robot swarms [5], packet routing [6], and autonomous driving [7].”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, background): “, 2016, 2017), autonomous driving (Shalev-Shwartz et al., 2016), and robotics (Kober and Peters, 2012).”
Artificial intelligence applications in the development of autonomous vehicles: a survey (2020, background): “Given many successful applications of DL approaches in other transportation areas such as traffic flow prediction, some researchers explored emerging AI approaches such as CNN, LSTM, and DBN [23], [65]–[67].”
On a Formal Model of Safe and Scalable Self-driving Cars (2017, methodology): “As noted in [5], even the Markovian assumption is not adequate for multi-agent scenarios, such as driving, and we will therefore later relax the assumption.”