Distributed agent-based air traffic flow management

By Kagan Tumer, Adrian K. Agogino. 2007

232 citations recorded by the source corpus.

Air traffic flow management is one of the fundamental challenges facing the Federal Aviation Administration (FAA) today. The FAA estimates that in 2005 alone, there were over 322,000 hours of delays at a cost to the industry in excess of three billion dollars. Finding reliable and adaptive solutions to the flow management problem is of paramount importance if the Next Generation Air Transportation Systems are to achieve the stated goal of accommodating three times the current traffic volume. This problem is particularly complex as it requires the integration and/or coordination of many factors including: new data (e.g., changing weather info), potentially conflicting priorities (e.g., different airlines), limited resources (e.g., air traffic controllers) and very heavy traffic volume (e.g., over 40,000 flights over the US airspace). In this paper we use FACET -- an air traffic flow simulator developed at NASA and used extensively by the FAA and industry -- to test a multi-agent algorithm for traffic flow management. An agent is associated with a fix (a specific location in 2D space) and its action consists of setting the separation required among the airplanes going though that fix. Agents use reinforcement learning to set this separation and their actions speed up or slow down traffic to manage congestion. Our FACET based results show that agents receiving personalized rewards reduce congestion by up to 45% over agents receiving a global reward and by up to 67% over a current industry approach (Monte Carlo estimation).

Subfields: Aerospace Engineering, Control and Systems Engineering, Economics and Econometrics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Control and Systems Engineering

In Economics and Econometrics

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, background): “The idea is inspired by difference rewards (Tumer & Agogino, 2007), in which each agent learns from a shaped reward that compares the global reward to the reward received when that agent’s action is replaced with a default action .”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “COMA [9] extends this approach with a counterfactual multi-agent critic baseline based on temporal difference errors [28] in order to facilitate multi-agent credit assignment.”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, methodology): “However, it has been applied with success in some applications (Sen, Sekaran, & Hale, 1994; Sen & Sekaran, 1998; Busoniu et al., 2006; Wang & de Silva, 2006; Tumer & Agogino, 2007, 2010).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “This counterfactual baseline has its roots in difference rewards, which is a method for obtaining the individual contribution of an agent in a cooperative multiagent team [332].”
Policy invariance under reward transformations for multi-objective reinforcement learning (2017, background): “framework, including air traffic control [12], traffic signal control [2], data routing in networks [13], electricity generator scheduling [14, 15], RoboCup soccer [16] and water resource management [17].”
Modeling difference rewards for multiagent learning (2012, methodology): “Following [3], we use the difference reward Di(z) = G(z) − G(z − zi) to provide the reward signal to each agent, where z − zi is a modified version of the normal action vector z in which agent i takes a “default” action (in our case, setting its “Miles in Trail” value to zero).”
Analyzing and visualizing multiagent rewards in dynamic and stochastic domains (2008, background): “It is important to note that though the particular projection method is domain dependent, the impact of factoredness and learnability on the reward is applicable to a broad array of domains [24,25,29].”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, methodology): “, 2017a) and the proposed solution was based on previous work in the area (Tumer and Agogino, 2007).”
Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks (2018, methodology): “based MARL, IL based MARL has been applied in some applications by choosing the proper exploration strategy such as in [27], [47].”
Design and control of compliant tensegrity robots through simulation and hardware validation (2014, background): “Each component can individually learn a control policy that decides how to actuate its individual end cap in such a way that global performance is maximized [26].”
A multiagent approach to managing air traffic flow (2010, background): “2 This agent-scaling results here are significantly different than those reported in [47] on similar data.”
A Review of the Applications of Agent Technology in Traffic and Transportation Systems (2010, background): “The individual vehicles are able to dispatch lightweight agents for exploring alternate routes and inform the road infrastructure about its travel intention.”
Potential-based difference rewards for multiagent reinforcement learning (2014, background): “The difference reward can generally be estimated, even in extremely complex domains like air traffic in the US airspace [14].”
A Theoretical and Empirical Analysis of Reward Transformations in Multi-Objective Stochastic Games (2017, background): “[2, 5, 10, 17]), but research into multi-objective applications is still at a very early stage.”