Safety Verification of Deep Neural Networks

By Xiaowei Huang, Marta Z. Kwiatkowska, Sen Wang, Min Wu. 2016

1,011 citations recorded by the source corpus.

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on safety of image classification decisions with respect to image manipulations, such as scratches or changes to camera angle or lighting conditions that would result in the same class being assigned by a human, and define safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.

Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability, Program Analysis & Verification

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. J. Kolter
  3. Cho-Jui Hsieh
  4. Huan Zhang
  5. Matthias Hein
  6. Nicholas Carlini
  7. Sergey Levine
  8. Pushmeet Kohli

In Hardware and Architecture

  1. Nicholas Carlini
  2. J. Kolter
  3. Eric Wong
  4. Suman Sekhar Jana
  5. Nicolas Papernot
  6. Ian J. Goodfellow
  7. Cho-Jui Hsieh
  8. Patrick Drew McDaniel

In Safety, Robustness & Interpretability

  1. Huan Zhang
  2. Cho-Jui Hsieh
  3. J. Kolter
  4. Nicholas Carlini
  5. Matthias Hein
  6. Eric Wong
  7. Suman Sekhar Jana
  8. Percy Liang

In Program Analysis & Verification

  1. Suman Sekhar Jana
  2. Eric Wong
  3. J. Kolter
  4. Huan Zhang
  5. Shiqi Wang
  6. Cho-Jui Hsieh
  7. Timon Gehr
  8. Clark W. Barrett

Citations in the corpus