Differential Adaptive Stress Testing of Airborne Collision Avoidance Systems

By Ritchie Lee, Mykel J. Kochenderfer, O. Mengshoel, Guillaume P. Brat, Michael P. Owen. 2018

88 citations recorded by the source corpus.

The next-generation Airborne Collision Avoidance System (ACAS X) is currently being developed and tested to replace the Traffic Alert and Collision Avoidance System (TCAS) as the next international standard for collision avoidance. To validate the safety of the system, stress testing in simulation is one of several approaches for analyzing nearmid-air collisions (NMACs). Understanding how NMACs can occur is important for characterizing risk and informing development of the system. Recently, adaptive stress testing (AST) has been proposed as a way to find the most likely path to a failure event. The simulation-based approach accelerates search by formulating stress testing as a sequential decision process then optimizing it using reinforcement learning. The approach has been successfully applied to stress test a prototype of ACAS X in various simulated aircraft encounters. In some applications, we are not as interested in the system’s absolute performance as its performance relative to another system. Such situations arise, for example, during regression testing or when deciding whether a new system should replace an existing system. In our collision avoidance application, we are interested in finding cases where ACAS X fails but TCAS succeeds in resolving a conflict. Existing approaches do not provide an efficient means to perform this type of analysis. This paper extends the AST approach to differential analysis by searching two simulators simultaneously and maximizing the difference between their outcomes. We call this approach differential adaptive stress testing (DAST). We apply DAST to compare a prototype of ACAS X against TCAS and show examples of encounters found by the algorithm.

Subfields: Aerospace Engineering, Automotive Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Adrian Weller
2. Ruth C. Fong
3. Yoshua Bengio
4. Mykel J. Kochenderfer
5. Peter Henderson

In Automotive Engineering

1. Adrian Weller
2. Mykel J. Kochenderfer

Citations in the corpus

• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, methodology): “For example, adaptive stress testing (AST) of an AI system allows users to find the most likely failure of a system for a given scenario using reinforcement learning [61], and is being used by to validate the next generation of aircraft collision avoidance software [62].”
• Survey on Scenario-Based Safety Assessment of Automated Vehicles (2020, background): “Both papers build on a predecessor paper [129] from the avionic domain.”
• The Adaptive Stress Testing Formulation (2020, background): “One possible approach to approximate validation is adaptive stress testing (AST) [7].”