Large Margin Deep Networks for Classification

By Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, Samy Bengio. 2018

328 citations recorded by the source corpus.

We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any norm on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques (such as weight decay, dropout, and batch norm).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Improving Neural Language Modeling via Adversarial Training (2019, background): “Direct Diversity Regularization There has been a body of literature on increasing the robustness by directly adding various forms of diversity-enforcing penalty functions (e.g., Elsayed et al., 2018; Xie et al., 2017; Liu et al., 2016; 2017; Chen et al., 2017; Wang et al., 2018).”
Supervised Contrastive Learning (2020, neutral): “A number of papers have studied other drawbacks of the cross-entropy loss, such as sensitivity to noisy labels [63, 46], presence of adversarial examples [10, 36], and poor margins [2].”
Fantastic Generalization Measures and Where to Find Them (2019, background): “Please see (Elsayed et al., 2018) and (Wei & Ma, 2019b) for more discussions. 2See Dziugaite & Roy (2017) for a non-vacuous generalization bound and related discussions.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “…cross-entropy loss, demonstrating that it leads to poor generalization performance due to poor margins (Liu et al., 2016; Cao et al., 2019), and lack of robustness to noisy labels (Zhang & Sabuncu, 2018; Sukhbaatar et al., 2015) or adversarial examples (Elsayed et al., 2018; Nar et al., 2019).”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “Recent theoretical and empirical analyses suggest that margin distributions are predictive of neural network generalization [8, 15, 26, 43].”
Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin (2019, methodology): “Prior works have studied, both empirically and theoretically, the margin of a network with respect to single perturbations at an intermediate or input layer (Sokolić et al., 2017; Novak et al., 2018; Elsayed et al., 2018; Jiang et al., 2018).”
On the Geometry of Adversarial Examples (2018, background): “There have been attempts to incorporate maximum margins into deep learning (Sun et al. (2016); Liu et al. (2016); Liang et al. (2017); Elsayed et al. (2018)), often by designing loss functions that encourage large margins at either the output (Sun et al. (2016)) or at any layer (Elsayed et al.…”
FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding (2021, background): “A common approach to learn well-separated decision boundary is to use a large margin classifier [24], but with our trials, category-level positive-margin based classifiers does not work in this data-hunger setting [20, 25].”
Analyzing and Improving Representations with the Soft Nearest Neighbor Loss (2019, background): “Indeed, this would translate to larger margins between different classes (Elsayed et al., 2018).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “The most used cross-entropy loss in text generation, i.e., the KL-divergence between one-hot vectors of labels and the distribution of model’s outputs, lacks robustness to noise labels [278] or adversarial examples [49].”
MMA Training: Direct Input Space Margin Maximization through Adversarial Training (2018, methodology): “First-order Large Margin: Previous works (Matyasko & Chau, 2017; Elsayed et al., 2018; Yan et al., 2019) have attempted to use ﬁrst-order approximation to estimate the input space margin.”
On the Connection Between Adversarial Robustness and Saliency Map Interpretability (2019, background): “Similar identities were previously also independently derived in (Elsayed et al., 2018) and (Jakubovitz & Giryes, 2018).”
Gradient penalty from a maximum margin perspective. (2020, result): “Adler and Lunz (2018) showed a similar result on dual norms.”
A Generalized Supervised Contrastive Learning Framework (2022, background): “In supervised context, cross-entropy is most widely used to train classifier networks, while several papers have studied the limitations of cross-entropy [48, 12, 3].”
Information-Theoretic Local Minima Characterization and Regularization (2019, background): “Similarly, Elsayed et al. (2018) employed a structured margin loss to improve performance of DNNs w.r.t. noise and adversarial attack yet no metric was proposed.”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, background): “Since CL methods rely extensively on cross-entropy loss for classiﬁcation tasks, they are prone to lack of robustness to noisy labels (Sukhbaatar et al., 2015) and the possibility of poor margins (Elsayed et al., 2018) affecting their ability to generalize across tasks.”
Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs (2019, methodology): “…to derive a pseudo-margin is to use Taylor’s approximation before solving the problem rather than after (as done by Matyasko and Chau (2017) and Elsayed et al. (2018)): γp(x) = min r ||r||p s.t. f(x+ r) = 0 ≈ min r ||r||p s.t. f(x) +∇xf(x)T r = 0 = |f(x)| ||∇xf(x)||q , where || · ||q is the…”