Robust Large Margin Deep Neural Networks

By Jure Sokolić, Raja Giryes, Guillermo Sapiro, Miguel Raul Dias Rodrigues. 2016

349 citations recorded by the source corpus.

The generalization error of deep neural networks via their classification margin is studied in this paper. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary nonlinearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization reparametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

A Closer Look at Memorization in Deep Networks (2017, background): “To achieve this goal, we build on the intuition that the number of different decision regions into which an input space is partitioned reflects the complexity of the learned hypothesis (Sokolic et al., 2016).”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “This phenomenon has received particular attention in the context of deep neural networks, and there is now a quickly growing body of work on this topic (Fawzi et al., 2015; Kurakin et al., 2016; Papernot & McDaniel, 2016; Rozsa et al., 2016; Torkamani, 2016; Sokolic et al., 2016; Tramèr et al., 2017”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “Distance-preserving neural networks and bi-Lipschitz condition The theoretic connection between distance preservation and the bi-Lipschitz condition is well-established [67], and learning an approximately isometric, distance-preserving transform has been an important goal in the fields of dimensiona”
Invertible Residual Networks (2018, background): “ks g = W 3˚(W 2˚(W 1)), where W i are convolutional layers. Hence, Lip(g) <1; if kW ik 2 <1; where kk 2 denotes the spectral norm. Note, that regularizing the spectral norm of the Jacobian of g(Sokoli et al., 2017) only reduces it locally and does not guarantee the above condition. Thus, we wi”
Sorting out Lipschitz function approximation (2018, methodology): “For classification tasks, a small Lipschitz constant leads to better generalization (Sokolić et al., 2017), improved adversarial robustness (Cisse et al.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, methodology): “The quantity ||Jx||F has been related to generalization both theoretically (Sokolić et al., 2017) and empirically (Novak et al., 2018).”
Generalization in Deep Learning (2017, methodology): “ﬁcation, if the margin is ensured to be large, we can ﬁx the partition with balls of radius corresponding to the large margin, ﬁlling the input space. Recently, this idea was applied fordeep learning(Sokolic et al., 2017a,b). Theseprevious results still suﬀer from the curse of (a priori known eﬀecti”
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021, methodology): “As an extension to recent methods for improving the model local smoothness (Miyato et al., 2018; Sokolić et al., 2017), PDR aims at regularizing the posterior difference between the clean and noisy inputs with regard to the family of f -divergences (Csiszár and Shields, 2004).”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, neutral): “…our results to MNNs with multi-output and multiple hidden layers, or combining our training error results with novel generalization bounds which might be better suited for MNNs ( e.g. , (Feng et al., 2016; Sokolic et al., 2016; Dziugaite & Roy, 2017)) than previous approaches (Zhang et al., 2017a”
Building a Regular Decision Boundary with Deep Networks (2017, background): “In [26], bounds on the margin are obtained with hypothesis of low dimensional structures, and 0.”
Deep Convolutional Framelet Denosing for Low-Dose CT via Wavelet Residual Network (2017, background): “This is true because the convolutional framelets consists of convolutional ﬁlters with ﬁnite coefﬁcients and ReLU, so the Jacobian can be represented as the product of the ﬁlter norms [49].”
Mathematics of Deep Learning (2017, background): “As it is hard to calculate the input margin directly, in [28] it is tied to the Jacobian matrix and the loss of the deep networks showing that bounding the spectral norm of the Jacobian matrix reduces the generalization error.”
The Lipschitz Constant of Self-Attention (2020, background): “, 2019), and guaranteed generalisation bounds (Sokolić et al., 2017).”
Procedural Noise Adversarial Examples for Black-Box Attacks on Deep Convolutional Networks (2018, background): “e hope future work explore more input-agnostic defences that minimize the sensitivity of large-scale models to small perturbations with techniques like model compression [20], Jacobian regularization [27, 30, 62, 69], or other types of denoising. 8 CONCLUSION We highlight the strengths of procedural”