Efficient Neural Network Robustness Certification with General Activation Functions

By Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, Luca Daniel. 2018

936 citations recorded by the source corpus.

Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some recent progress has been made towards this direction by exploiting the piece-wise linear nature of ReLU activations. However, a generic robustness certification for general activation functions still remains largely unexplored. To address this issue, in this paper we introduce CROWN, a general framework to certify robustness of neural networks with general activation functions for given input data points. The novelty in our algorithm consists of bounding a given activation function with linear and quadratic functions, hence allowing it to tackle general activation functions including but not limited to four popular choices: ReLU, tanh, sigmoid and arctan. In addition, we facilitate the search for a tighter certified lower bound by adaptively selecting appropriate surrogates for each neuron activation. Experimental results show that CROWN on ReLU networks can notably improve the certified lower bounds compared to the current state-of-the-art algorithm Fast-Lin, while having comparable computational efficiency. Furthermore, CROWN also demonstrates its effectiveness and flexibility on networks with general activation functions, including tanh, sigmoid and arctan.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “…smoothing to other certifiably robust classifiers for `2 norm that have been proposed in the literature (Wong et al., 2018; Tsuzuku et al., 2018; Zhang et al., 2018), and find that randomized smoothing outperforms all of them by a wide margin.2 Finally, we use randomized smoothing to construct…”
Efficient Formal Safety Analysis of Neural Networks (2018, background): “Also, their follow-up work [44] that provides a generic relaxation method for general activation functions does not address this issue either.”
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers (2019, background): “On the bright side, these methods are more scalable and efficient which makes some of them useful for building certified defenses [38, 35, 36, 27, 28, 39, 10, 9, 7, 13, 26, 30, 15, 37, 40].”
A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks (2019, methodology): “Other activation functions can be linearly bounded as discussed in CROWN (Zhang et al., 2018), DeepZ and DeepPoly (Singh et al., 2019b); CROWN and DeepPoly are also more general and do not require a i to allow a more ﬂexible selection of bounds.”
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks (2019, background): “This can be done either locally around a specific input [7, 35, 15, 33, 12, 29, 30, 13, 40, 21, 41, 42], or globally by bounding the sensitivity of the function to input perturbations, i.”
Certified Adversarial Robustness with Additive Noise (2018, background): “Recently, their idea has been extended by [15] and relaxed to general activation functions.”
Trustworthy AI: From Principles to Practices (2021, background): “For example, CNN-Cert [51], CROWN [379], Fast-lin, and Fast-lip [352] aim to minimize an upper bound of the worst-case loss under given input perturbations.”
Lipschitz-Certifiable Training with a Tight Outer Bound (2020, methodology): “Thus, many veriﬁcation methods are proposed to guarantee stable prediction of input within a perturbation set [18, 5, 9, 16, 38, 34, 23, 11, 3, 30, 8, 43, 2].”
Towards Stable and Efficient Training of Verifiably Robust Neural Networks (2019, methodology): “, 2018) and CROWN (Zhang et al., 2018), the verified errors increase significantly.”
Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond (2020, methodology): “To compute LiRPA bounds, the first step is to obtain linear relaxations of any non-linear units [53, 35] (e.”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “Convex relaxation of non-linear units in neural networks enables an efficient analysis of the outer bounds for a neural network [79, 86, 66, 13, 78, 76, 57, 67].”
Logit Pairing Methods Can Fool Gradient-Based Attacks (2018, background): “Thus, providing non-trivial lower bounds – with or without special training – is an important and active area of research [Wong et al., 2018, Zhang et al., 2018, Xiao et al., 2018, Croce et al., 2018].”
Convergence of Adversarial Training in Overparametrized Neural Networks (2019, neutral): “In contrast to attack algorithms, neural network veriﬁcation methods [48, 47, 54, 42, 14, 38] tries to ﬁnd upper bounds of the robust loss and provide certiﬁed robustness measurements.”
Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes (2019, background): “There has been also been some recent work lower bounding the pointwise robustness in the `2 norm (Zhang et al., 2018; Wong et al., 2018b; Weng et al., 2018) .”
Exactly Computing the Local Lipschitz Constant of ReLU Networks (2020, background): “In addition, many convex-relaxation approaches towards certiﬁable robustness may be recovered by this framework (Zico Kolter & Wong, 2017; Raghunathan et al., 2018; Zhang et al., 2018; Singh et al., 2019).”
Relating Adversarially Robust Generalization to Flat Minima (2021, background): “Besides certified and provable defenses [22, 138, 66, 147, 146, 130, 41, 39, 84, 113, 72, 23], adversarial training (AT) has become the de-facto standard, as discussed in the main paper.”