DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks

By Divya Gopinath, Guy Katz, Corina S. Pasareanu, Clark W. Barrett. 2017

93 citations recorded by the source corpus.

Deep neural networks have become widely used, obtaining remarkable results in domains such as computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, and bio-informatics, where they have produced results comparable to human experts. However, these networks can be easily fooled by adversarial perturbations: minimal changes to correctly-classified inputs, that cause the network to mis-classify them. This phenomenon represents a concern for both safety and security, but it is currently unclear how to measure a network's robustness against such perturbations. Existing techniques are limited to checking robustness around a few individual input points, providing only very limited guarantees. We propose a novel approach for automatically identifying safe regions of the input space, within which the network is robust against adversarial perturbations. The approach is data-guided, relying on clustering to identify well-defined geometric regions as candidate safe regions. We then utilize verification techniques to confirm that these regions are safe or to provide counter-examples showing that they are not safe. We also introduce the notion of targeted robustness which, for a given target label and region, ensures that a NN does not map any input in the region to the target label. We evaluated our technique on the MNIST dataset and on a neural network implementation of a controller for the next-generation Airborne Collision Avoidance System for unmanned aircraft (ACAS Xu). For these networks, our approach identified multiple regions which were completely safe as well as some which were only safe for specific labels. It also discovered several adversarial perturbations of interest.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems (2018, background): “Moreover, DeepSafe [19] focuses on automatically identifying safe regions of the input space, within which the network is robust against adversarial perturbations.”
Secure and Robust Machine Learning for Healthcare: A Survey (2020, background): “adversarial and original samples that can be regarded as the detector model [159], [160].”
Feature-Guided Black-Box Safety Testing of Deep Neural Networks (2017, methodology): “r programming is extended to work with the piecewise linear ReLU functions that cannot be expressed using linear programming. The approach can scale up to networks with 300 ReLU nodes. In recent work [14] the input vector space is partitioned using clustering and then the method of [17] is used to c”
Software Engineering for AI-Based Systems: A Survey (2021, background): “These approaches explore space of adversarial counter-examples to identify and ensure safe regions of the input space, within which the network is robust against adversarial perturbations [70].”
Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for L0 Norm (2018, methodology): “In recent work [6] the input vector space is partitioned using clustering and then the method of [9] is used to check the individual partitions.”
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks (2019, methodology): “DeepSafe [18] tries to identify safe regions in the input space using Reluplex as its core.”
The Marabou Framework for Verification and Analysis of Deep Neural Networks (2019, methodology): “The Marabou project builds upon our previous work on the Reluplex project [2,7,12,13,15,17], which focused on applying SMT-based techniques to the veriﬁcation of DNNs. Marabou follows the Reluplex spirit in that it applies an SMT-based, lazy search technique: it iteratively searches for an…”
DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems (2018, methodology): “DeepSafe [22] uses Reluplex as its verification engine and has the same scalability problem as Reluplex.”