Large-Margin Softmax Loss for Convolutional Neural Networks

By Weiyang Liu, Yandong Wen, Zhiding Yu, Meng Yang. 2016

1,593 citations recorded by the source corpus.

Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a generalized large-margin softmax (L-Softmax) loss which explicitly encourages intra-class compactness and inter-class separability between learned features. Moreover, L-Softmax not only can adjust the desired margin but also can avoid overfitting. We also show that the L-Softmax loss can be optimized by typical stochastic gradient descent. Extensive experiments on four benchmark datasets demonstrate that the deeply-learned features with L-softmax loss become more discriminative, hence significantly boosting the performance on a variety of visual classification and verification tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “With this goal, numerous works have pushed the image recognition performance from different directions, such as data scale from MNIST [36] to ImageNet1K [10], model architectures from convolutional neural networks (CNNs) [23, 26, 34, 35, 40, 52, 53] to Transformers [15, 43, 57, 61, 64, 67, 72], and ”
Recent advances in convolutional neural networks (2015, methodology): “In this section, we introduce some fast processing methods of CNNs.”
Improving Neural Language Modeling via Adversarial Training (2019, background): “Direct Diversity Regularization There has been a body of literature on increasing the robustness by directly adding various forms of diversity-enforcing penalty functions (e.g., Elsayed et al., 2018; Xie et al., 2017; Liu et al., 2016; 2017; Chen et al., 2017; Wang et al., 2018).”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “Then we can write the large-margin cross-entropy loss (Liu et al., 2016) as LLarge-Margin = ∑ i exp ( ‖wyi‖ · ‖xi‖ · cos(θyi + ∆θ) )∑ j exp ( ‖wj‖ · ‖xi‖ · cos(θj + 1(j = yi) ·∆θ) ) (11) where we have that ∆θ = (m− 1)θ.”
Forward Compatible Few-Shot Class-Incremental Learning (2022, background): “For example, [14] seeks to detect new classes by learning placeholders, [5] utilizes the embedding with large margin between classes, [7] encourages class-wise orthogonality for more compact embedding.”
SphereFace: Deep Hypersphere Embedding for Face Recognition (2017, methodology): “For fair comparison, we also implement the softmax loss, contrastive loss, center loss, triplet loss, L-Softmax loss [16] and train them with the same 64-layer CNN architecture as A-Softmax loss.”
Supervised Contrastive Learning (2020, neutral): “A number of works have explored shortcomings of this loss, such as lack of robustness to noisy labels [63, 46] and the possibility of poor margins [10, 31], leading to reduced generalization performance.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “Cross entropy loss leads to poor generalization performance due to poor margins (Liu et al., 2016; Cao et al., 2019), and it lacks robustness to noisy labels (Zhang & Sabuncu, 2018; Sukhbaatar et al., 2015) or adversarial examples (Elsayed et al., 2018; Nar et al., 2019).”
CosFace: Large Margin Cosine Loss for Deep Face Recognition (2018, background): “[24] proposed a large margin softmax (L-Softmax) by adding angular constraints to each identity to improve feature discrimination.”
Multi-label Contrastive Predictive Coding (2020, methodology): “It would be interesting to further apply this method to other application domains, investigate alternative methods to control the re-weighting procedure (such as using angular margins [33]), and develop more principled approaches towards curriculum learning for unsupervised representation learning.”
Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review (2017, methodology): “More recently, the research in [74] proposed a Large-Margin Softmax Loss (L-softmax), which is a modification of the softmax loss with a distance margin constraint.”
Regularizing Neural Networks via Minimizing Hyperspherical Energy (2019, background): “The importance of regularizing angular information is also discussed in [31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
Unequal-Training for Deep Face Recognition With Long-Tailed Noisy Data (2019, background): “A more powerful type of loss function is large margin softmax loss, mainly containing SphereFace [21](L-Softmax [22]), CosFace [36] and Ar-cFace [7], which signiﬁcantly boost the face recognition.”
Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation (2017, methodology): “Inspired by [39, 31], we design the architectures as shown in Table 2.”
Improving Lexical Choice in Neural Machine Translation (2017, result): “Recent work by Liu et al. (2016) has a similar motivation to that of our ﬁxnorm model.”
Dense Classification and Implanting for Few-Shot Learning (2019, methodology): “Multiple improvements of the standard softmax and cross-entropy loss are proposed by [48, 22, 52, 44, 9] to this end.”
Hyperspherical Prototype Networks (2019, neutral): “[23], for example, improve the separation in softmax cross-entropy by increasing the angular margin between classes.”
Deep Imbalanced Learning for Face Recognition and Attribute Prediction (2018, background): “Beijing FaceAll_Norm_1600 Large 64.80 67.11 Google - FaceNet v8 Large 70.49 86.47 NTechLAB - facenx_large Large 73.30 85.08 SIATMMLAB TencentVision Large 74.20 87.27 DeepSense V2 Large 81.29 95.99 YouTu Lab Large 83.29 91.34 Vocord - deepVo V3 Large 91.76 94.96 SIAT_MMLAB Small 65.23 76.72 DeepSense”
Long-tail learning via logit adjustment (2020, background): “See also Liu et al. [2016, 2017], Wang et al. [2018], Khan et al. [2019] for similar weighting of negatives in the softmax.”