L2-Nonexpansive Neural Networks

By Haifeng Qian, Mark N. Wegman. 2018

75 citations recorded by the source corpus.

This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers. We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness, with a new regularization scheme for linear layers, new ways to adapt nonlinearities and a new loss function. With MNIST and CIFAR-10 classifiers, we demonstrate a number of advantages. Without needing any adversarial training, the proposed classifiers exceed the state of the art in robustness against white-box L2-bounded adversarial attacks. They generalize better than ordinary networks from noisy data with partially random labels. Their outputs are quantitatively meaningful and indicate levels of confidence and generalization, among other desirable properties.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Adversarial Training for Free! (2019, background): “, 2017], L2-nonexpansive nets [Qian and Wegman, 2018], Jacobian Regularization [Jakubovitz and Giryes, 2018], Universal Perturbation [Shafahi et al.”
On Evaluating Adversarial Robustness (2019, background): “For example, in some cases even a million iterations of white-box gradient descent have been found necessary (Qian & Wegman, 2018).”
Diametrical Risk Minimization: theory and computations (2019, result): “…et al., 2017) on the magnitude of the Lipschitz moduli, as well as the growing support for the sharp-vs-flat hypothesis, have even given rise to research centered on Lipschitz regularization (Gouk et al., 2018; Oberman & Calder, 2018; Qian & Wegman, 2018) for improving the generalization of NNs.”
A Closer Look at Accuracy vs. Robustness (2020, background): “For neural networks, Anil et al. [2], Qian and Wegman [39], Huster et al. [22] provide methods for imposing global Lipschitzness constraints; however, the state-of-the-art methods for training such networks do not lead to highly expressible functions.”
Defensive Quantization: When Efficiency Meets Robustness (2019, background): “Recent works (Cisse et al., 2017; Qian & Wegman, 2018) have attempted to make the network non-expansive by controlling the Lipschitz constant of the network to be smaller than 1, which has smaller variation change in its output than its input.”
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle (2019, methodology): “To improve the robustness of neural networks to adversarial examples, many defense strategies and models have been proposed, such as adversarial training [25], orthogonal regularization [6,22], L2 nonexpansive networks [28], Bayesian method [41], random masking [24], TRADES [42], rejecting adversari”
Limitations of the Lipschitz constant as a defense against adversarial examples (2018, background): “[4] and [12] enforce constraints on the singular values of matrices as a way of increasing robustness to existing attacks.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “…a linear layer can be controlled/bounded during training by normalizing the weight with (approximate) orthogonality [23], [24], which is an important property for certiﬁed defense against adversarial attacks [282], [283], [284], and for theoretically analyzing DNNs generalization [285], [286].”
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers (2019, background): “Regularizing Lipschitz continuity stabilizes training, improves the ﬁnal performance, and enhances robustness to adversarial attacks (Weng et al., 2018; Qian & Wegman, 2019).”
Testing Robustness Against Unforeseen Adversaries (2019, background): “While evaluating with moderately many PGD steps (200) helps guard against this, [31] shows that an L∞-trained model that appeared robust against L2 actually had substantially less robustness when evaluating with 10(6) PGD steps.”
Adversarially Robust Neural Architectures (2020, unknown): “turbation budget is set to 8=255 for CIFAR-10/100 and 76:5=255 for MNIST. Our baseline is PGD-AT since the searched cells are PGD-adversarially retrained as in [14]. RACL is also compared with L2NNN [23] which proposes to control Lipschitz constant to maximize conﬁdence gaps, ADP [29] which introduc”
You Only Propagate Once: Painless Adversarial Training Using Maximal Principle (2019, methodology): “To improve the robustness of neural networks to adversarial examples, many defense approaches have been proposed, like, adversarial trainining [5], orthogonal regularization [10,11], L2 nonexpansive networks [12], Bayesian method [13], random masking [14], TRADES [7], rejecting adversarial examples ”