Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control

By Tianshu Chu, Jie Wang, Lara Codecà, Zhaojian Li. IEEE transactions on intelligent transportation systems (Print), 2019

870 citations recorded by the source corpus.

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, the centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. The multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now, the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent, advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. The results demonstrate its optimality, robustness, and sample efficiency over the other state-of-the-art decentralized MARL algorithms.

Subfields: Building and Construction, Control and Systems Engineering, Transportation

View Paper: https://www.semanticscholar.org/paper/b17e0b5b5c7e632ecf5f35b0616cbf820382fcf7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Building and Construction

In Control and Systems Engineering

In Transportation

Citations in the corpus

Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey (2020, background): “[102] A2C-RNN Policy sharing No 5 by 5 grid Monaco city map SUMO Ind-Q-learning Ind-DQN Ind-A2C”
A Survey on Traffic Signal Control Methods (2019, methodology): “ing et al. 2004a,b] Model-free methods [Abdoos et al. 2011a, 2014; Arel et al. 2010; Aslani et al. 2017, 2018b; Bakker et al. 2010; Balaji et al. 2010; Brys et al. 2014; Casas 2017; Chin et al. 2011; Chu et al. 2019; El-Tantawy and Abdulhai 2010, 2012; El-Tantawy et al. 2013; Iša et al. 2006; Mannio”
Dynamic and Multi-faceted Spatio-temporal Deep Learning for Traffic Speed Forecasting (2021, background): “The successful applications include but not limited to computer vision methods for traffic data collection [9] and regulations enforcement [2], reinforcement learning for automatic pilot [16, 18] and traffic signal control [6], big data analysis for human mobility modelling [10, 11], etc.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “transportation schedule [22, 53, 159, 253], traffic signal control [50, 59], energy management [180], wireless security [172], satellite docking [76], edge computing [299], chemical processes [241], video games [28, 153, 192, 211, 262, 300], board games of Go, shogi, chess and arcade game PAC-MAN [1”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “For example, [39] show that sharing local information could be very helpful while keeping the algorithm practical.”
Graph Neural Networks for Intelligent Transportation Systems: A Survey (2023, methodology): “Finally, they compared the framework with baselines (fixed time [158], Max Pressure [159], CGRL [160], Individual RL [161], OneModel [162], Neighbor RL [163], GCN [155], and Co-Light [157]) on two real datasets, and concluded that their model outperforms the state-of-the-art.”
CoLight: Learning Network-level Cooperation for Traffic Signal Control (2019, methodology): “56 OneModel [5] 181.”