Integrating Independent and Centralized Multi-agent Reinforcement Learning for Traffic Signal Network Optimization
By Zhi Zhang, Jiachen Yang, Hongyuan Zha. 2019
48 citations recorded by the source corpus.
Traffic congestion in metropolitan areas is a world-wide problem that can be ameliorated by traffic lights that respond dynamically to real-time conditions. Recent studies that applied deep reinforcement learning (RL) to optimize single traffic lights have shown significant improvement over conventional control. However, optimization of global traffic flow over a large road network fundamentally is a cooperative multi-agent control problem. Centralized learning via single-agent RL is infeasible due to an exponential joint-action space, while independent learning suffers from environment non-stationarity. We propose QCOMBO, a simple yet effective multi-agent reinforcement learning (MARL) algorithm that combines the advantages of independent and centralized learning without their shortcomings. We ensure scalability by selecting actions from individually optimized utility functions, which are shaped to maximize global performance via a novel consistency regularization loss between individual utility and a global action-value function. Experiments on diverse road topologies and traffic flow conditions in the SUMO traffic simulator show competitive performance of QCOMBO versus recent state-of-the-art MARL algorithms. We further show that policies trained on small sub-networks can effectively generalize to larger networks under different traffic flow conditions, providing empirical evidence for the suitability of MARL for intelligent traffic control.
Subfields: Building and Construction, Control and Systems Engineering, Transportation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Control and Systems Engineering
Citations in the corpus
- FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “CTDE can resolve the non-stationarity under partial observability and has demonstrated great potential to address complex real-world problems, such as traffic signal control (Zhang et al., 2020; Xu et al., 2021) and autonomous driving cars (Zhou et al., 2020).”
- Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “progress in many domains, including traffic signal network optimization [7, 31, 42] and network packet routing [22].”