Guiding Deep Learning System Testing Using Surprise Adequacy

By Jinhan Kim, Robert Feldt, Shin Yoo. International Conference on Software Engineering, 2018

496 citations recorded by the source corpus.

Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

Subfields: Artificial Intelligence, Software, Signal Processing

View Paper: https://www.semanticscholar.org/paper/851ec332236952a37236d229521584963f28d150

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaowei Huang
2. Wenjie Ruan
3. Daniel Kroening
4. Youcheng Sun
5. Min Wu
6. Yang Liu
7. James Sharp
8. Jie M. Zhang

In Software

1. Jie M. Zhang
2. Yang Liu
3. Lei Ma
4. Daniel Kroening
5. Mark Harman
6. Xinping Yi
7. James Sharp
8. Xiaowei Huang

In Signal Processing

1. Mark Harman
2. Xiaowei Huang
3. Lei Ma
4. Min Wu
5. Yang Liu
6. Jie M. Zhang

Citations in the corpus

• A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability (2018, methodology): “In terms of test case generation, [Kim et al., 2018b] utilises a few existing algorithms for adversarial attack, including FGSM [Goodfellow et al., 2014b], Basic iterative method [Kurakin et al., 2016], JSMA [Papernot et al., 2016c], CW attack [Carlini and Wagner, 2017a].”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “[127] proposed two measurements to evaluate the skew between training and test data: one is based on Kernel Density Estimation (KDE) to approximate the likelihood of the system having seen a similar input during training, the other is based on the distance between vectors representing the neuron act”