Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

By Binghui Chen, Weihong Deng, Junping Du. 2017

132 citations recorded by the source corpus.

Over the past few years, softmax and SGD have become a commonly used component and the default training strategy in CNN frameworks, respectively. However, when optimizing CNNs with SGD, the saturation behavior behind softmax always gives us an illusion of training well and then is omitted. In this paper, we first emphasize that the early saturation behavior of softmax will impede the exploration of SGD, which sometimes is a reason for model converging at a bad local-minima, then propose Noisy Softmax to mitigating this early saturation issue by injecting annealed noise in softmax during each iteration. This operation based on noise injection aims at postponing the early saturation and further bringing continuous gradients propagation so as to significantly encourage SGD solver to be more exploratory and help to find a better local-minima. This paper empirically verifies the superiority of the early softmax desaturation, and our method indeed improves the generalization ability of CNN model by regularization. We experimentally find that this early desaturation helps optimization in many tasks, yielding state-of-the-art or competitive results on several popular benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving Neural Language Modeling via Adversarial Training (2019, background): “Direct Diversity Regularization There has been a body of literature on increasing the robustness by directly adding various forms of diversity-enforcing penalty functions (e.g., Elsayed et al., 2018; Xie et al., 2017; Liu et al., 2016; 2017; Chen et al., 2017; Wang et al., 2018).”
Sigsoftmax: Reanalysis of the Softmax Bottleneck (2018, background): “Even though softmax is widely used, few studies have attempted to improve its modeling performance [6, 8].”
Deep Imbalanced Learning for Face Recognition and Attribute Prediction (2018, unknown): “98 Noisy Softmax [66] 1 WebFace+ 99.”
Striking the Right Balance With Uncertainty (2019, unknown): “DeepFace [48] 97.35 91.4 FaceNet [42] 99.63 95.4 Web-scale [49] 98.37 - DeepID2+ [47] 99.47 93.2 Baidu [29] 99.13 - Center Face [55] 99.28 94.9 Marginal Loss [13] 99.48 96.0 Noisy Softmax [8] 99.18 94.9”
Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments (2017, methodology): “ts on face veriﬁcation task of LFW benchmark. Due to the good performance, we apply two well-published convolutional neural networks to compare LFW and CALFW, they are VGG-Face [11] and Noisy Softmax [4]. The parameters of two networks are set according to the original papersandweapplythenetworkmode”
Past, Present, and Future of Face Recognition: A Review (2020, background): “activation [143] and various loss functions [15, 124-126, 130, 134, 136, 138, 141-142, 145, 148-152], the”