Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge

By Pin Wang, Ching-yao Chan. 2017

124 citations recorded by the source corpus.

Multiple automakers have in development or in production automated driving systems (ADS) that offer freeway-pilot functions. This type of ADS is typically limited to restricted-access freeways only, that is, the transition from manual to automated modes takes place only after the ramp merging process is completed manually. One major challenge to extend the automation to ramp merging is that the automated vehicle needs to incorporate and optimize long-term objectives (e.g. successful and smooth merge) when near-term actions must be safely executed. Moreover, the merging process involves interactions with other vehicles whose behaviors are sometimes hard to predict but may influence the merging vehicle's optimal actions. To tackle such a complicated control problem, we propose to apply Deep Reinforcement Learning (DRL) techniques for finding an optimal driving policy by maximizing the long-term reward in an interactive environment. Specifically, we apply a Long Short-Term Memory (LSTM) architecture to model the interactive environment, from which an internal state containing historical driving information is conveyed to a Deep Q-Network (DQN). The DQN is used to approximate the Q-function, which takes the internal state as input and generates Q-values as output for action selection. With this DRL architecture, the historical impact of interactive environment on the long-term reward can be captured and taken into account for deciding the optimal control policy. The proposed architecture has the potential to be extended and applied to other autonomous driving scenarios such as driving through a complex intersection or changing lanes under varying traffic flow conditions.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Intersections/Merging Ego-vehicle required to negotiate intersections and merges into highways [86], Ramp merging is tackled in [87], where DRL is applied to ﬁnd an optimal driving policy using LSTM for producing an internal state containing historical driving information and DQN for Q-function…”
A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers (2018, methodology): “In one previous study [14], we used RL to train the vehicle agent to learn an optimal ramp-merge driving policy under interactive driving environment.”
Distributed Multiagent Coordinated Learning for Autonomous Driving in Highways Based on Dynamic Coordination Graphs (2020, unknown): “RL to realize motion planning at interactions [14] and ramp merging [38].”
Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles (2020, methodology): “In a merging scenario, the most crucial information is the relative longitudinal position and speed 2 x ( dx, dv ) of the two vehicles bounding the target gap, as used by [69].”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, methodology): “The AD problem of ramp merging is tackled in (Wang and Chan, 2017), where DRL is applied to ﬁnd an optimal driving policy using LSTM for producing an internal state containing historical driving information and DQN for Q-function approximation.”
A Survey of State-Action Representations for Autonomous Driving (2018, methodology): “This representation is used in (Coulom, 2002) for car racing, in (Paxton et al., 2017; Wang and Chan, 2017) for highway driving scenarios, and in (Shalev-Shwartz et al., 2016a) for a roundabout insertion scenario.”