DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems

By Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, Sarfraz Khurshid. International Conference on Automated Software Engineering, 2018

648 citations recorded by the source corpus.

While Deep Neural Networks (DNNs) have established the fundamentals of image-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To address the safety issues in autonomous driving systems, a recent set of testing techniques have been designed to automatically generate artificial driving scenes to enrich test suite, e.g., generating new input images transformed from the original ones. However, these techniques are insufficient due to two limitations: first, many such synthetic images often lack diversity of driving scenes, and hence compromise the resulting efficacy and reliability. Second, for machine-learning-based systems, a mismatch between training and application domain can dramatically degrade system accuracy, such that it is necessary to validate inputs for improving system robustness. In this paper, we propose DeepRoad, an unsupervised DNN-based framework for automatically testing the consistency of DNN-based autonomous driving systems and online validation. First, DeepRoad automatically synthesizes large amounts of diverse driving scenes without using image transformation rules (e.g. scale, shear and rotation). In particular, DeepRoad is able to produce driving scenes with various weather conditions (including those with rather extreme conditions) by applying Generative Adversarial Networks (GANs) along with the corresponding real-world weather scenes. Second, DeepRoad utilizes metamorphic testing techniques to check the consistency of such systems using synthetic images. Third, DeepRoad validates input images for DNN-based systems by measuring the distance of the input and training images using their VGGNet features. We implement DeepRoad to test three well-recognized DNN-based autonomous driving systems in Udacity self-driving car challenge. The experimental results demonstrate that DeepRoad can detect thousands of inconsistent behaviors for these systems, and effectively validate input images to potentially enhance the system robustness as well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ab758c7d163dba039f1b1badaa9ea72064c887ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Liu
2. Bo Li
3. Bowen Zhou
4. Jinfeng Yi
5. Percy Liang
6. Felix Juefei-Xu
7. Lei Ma
8. Simon See

In Computer Vision and Pattern Recognition

1. Yang Liu
2. Felix Juefei-Xu
3. Bo Li
4. Lei Ma
5. Simon See
6. Bowen Zhou
7. Chun Xia
8. Jianjun Zhao

Citations in the corpus

• DeepHunter: a coverage-guided fuzz testing framework for deep neural networks (2019, background): “DeepRoad [51] proposes a GAN-based scene mutation transformation ( i.e ., snowy and rainy) to test autonomous driving system.”
• Model Agnostic Defence Against Backdoor Attacks in Machine Learning (2019, methodology): “techniques, such as differential testing [11], metamorphic testing [16] [22] , concolic testing [13], model mutation [19] and probabilistic search [18].”
• Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, background): “DL libraries (e.g., TensorFlow [66] and PyTorch [57]) serve as the fundamental building block for all DL pipelines by providing thousands of APIs for building, training, and deploying DL models.”
• TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search (2020, background): “, testing [9–15], fault localization [16–18], program repair and synthesis [19–22], in multiple so‰ware development domains [23–25].”
• Trustworthy AI: From Principles to Practices (2021, methodology): “For example, in the field of autonomous driving, besides developing a perceptual system working in sunny scenes, academia and the industry are using numerous development and testing strategies to enhance the perceptual performance of the vehicles in nighttime/rainy scenes to guarantee the system’s r”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “Their work pioneered to propose coverage criteria for DNN. Enlightened by this paper, a number of machine learning testing techniques have emerged, such as DeepTest [76], DeepGauge [92], DeepConcolic [111], and DeepRoad [79].”