Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

By Yusuke Tsuzuku, Issei Sato, Masashi Sugiyama. 2018

377 citations recorded by the source corpus.

High sensitivity of neural networks against malicious perturbations on inputs causes security concerns. To take a steady step towards robust classifiers, we aim to create neural network models provably defended from perturbations. Prior certification work requires strong assumptions on network structures and massive computational costs, and thus the range of their applications was limited. From the relationship between the Lipschitz constants and prediction margins, we present a computationally efficient calculation technique to lower-bound the size of adversarial perturbations that can deceive networks, and that is widely applicable to various complicated networks. Moreover, we propose an efficient training procedure that robustifies networks and significantly improves the provably guarded areas around data points. In experimental evaluations, our method showed its ability to provide a non-trivial guarantee and enhance robustness for even large networks.

Subfields: Statistical and Nonlinear Physics, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Distance-preserving neural networks and bi-Lipschitz condition The theoretic connection between distance preservation and the bi-Lipschitz condition is well-established [67], and learning an approximately isometric, distance-preserving transform has been an important goal in the fields of dimensiona”
Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “For Tsuzuku et al. (2018), we computed the certified radius on every image in the SVHN test set; for randomized smoothing, we took every 40th image, for 651 images total.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “Many current approaches to Adversarial Learning similarly operate either by minimizing L at the global or local level [46, 73, 45] or by devising models characterized by a small L by design [22, 33].”
Invertible Residual Networks (2018, background): “Note, that a power-iteration on the parameter matrix [19] only gives a bound on (cid:107) W i (cid:107) 2 , see [22].”
Sorting out Lipschitz function approximation (2018, background): “[50], we can guarantee adversarial robustness at a point x by considering the margin about that point divided by the Lipschitz constant.”
Improving MMD-GAN Training with Repulsive Loss Function (2018, background): “However, compared to PIM, PICO has a higher computational cost which roughly equals the additional cost incurred by increasing the batch size by two (Tsuzuku et al. (2018)).”
Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes (2019, methodology): “Exactly computing the Lipschitz constant of neural networks has also been shown to be NP-hard [22], but overestimations of the Lipschitz constant have been shown to provide lower bounds to the pointwise robustness [15, 25, 19, 10, 21].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “a linear layer can be controlled/bounded during training by normalizing the weight with (approximate) orthogonality [23], [24], which is an important property for certified defense against adversarial attacks [282], [283], [284], and for theoretically analyzing DNNs generalization [285], [286].”
Lipschitz-Certifiable Training with a Tight Outer Bound (2020, methodology): “In contrast, layer-wise propagation approach proposed the worst-case logit or the certiﬁable margin in the logit space to obtain a differentiable upper bound on the robust classiﬁcation error [36, 13, 44].”
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks (2019, methodology): “Furthermore, Lipschitz bounds can be utilized during training as a heuristic to promote out-of-sample generalization [36].”
The Lipschitz Constant of Self-Attention (2020, background): “Lipschitz properties have been shown to improve robustness against adversarial perturbations (Cisse et al., 2017; Tsuzuku et al., 2018; Anil et al., 2019) and aid generalisation (Sokolić et al.”
Attacks Which Do Not Kill Training Make Adversarial Learning Stronger (2020, neutral): “, `2 and `∞) perturbations (Cohen et al., 2019; Wong & Kolter, 2018; Tsuzuku et al., 2018; Lécuyer et al., 2019; Weng et al., 2018; Balunovic & Vechev, 2020; Zhang et al., 2020).”
Limitations of the Lipschitz constant as a defense against adversarial examples (2018, methodology): “Finally, [16] penalizes the spectral norms of matrices and uses equation 6 to compute a Lipschitz constant for the network.”
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers (2019, background): “This relaxation underestimates the true spectral norm of the convolutional operator (Corollary 1 of (Tsuzuku et al., 2018)) given by σ(Kl) = max x 6=0 ‖Kl ∗ x‖2/‖x‖2, where x is the input to the convolutional layer and ∗ is the convolutional operator.”
Adversarial Training Versus Weight Decay (2018, background): “…just like rescaling the weight vector of a linear classiﬁer does not affect the hyperplane boundary it deﬁnes, varying the Lipschitz constant of a neural network by rescaling its output affects neither its predictions nor its vulnerability to adversarial perturbations (Tsuzuku et al., 2018).”