Self-Normalizing Neural Networks

By Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter. 2017

2,892 citations recorded by the source corpus.

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: this http URL.

Subfields: Artificial Intelligence, Materials Chemistry, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Model Architectures

In Optimization & Training

Citations in the corpus

Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, background): “However batch normalization introduces ﬂuctuations of normalizing constants which can be harmful [26, 15].”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “Deep learning has enabled signiﬁcant improvements in state-of-the-art performance across domains [1, 2, 3, 4].”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “The activation functions that we explored are: (1) GeLU (Hendrycks and Gimpel, 2016), (2) Swish (Ramachandran et al., 2017), (3) Exponential Linear Units (ELU) (Clevert et al., 2015), (4) Scaled exponential linear units (SeLU) (Klambauer et al., 2017), (5) Sigmoid and (6) Softplus.”
Swish: a Self-Gated Activation Function (2017, background): “While much work has focused on proposing new activation functions (Maas et al., 2013; Agostinelli et al., 2014; He et al., 2015; Clevert et al., 2015; Klambauer et al., 2017; Qiu & Cai, 2017), few studies, such as Xu et al. (2015), have systematically compared different activation functions.”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, neutral): “4) [49].”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “Scaled exponential linear unit (SELU) [68] introduce self-normalization in the activation function.”
Methods for interpreting and understanding deep neural networks (2017, background): “Examples are LSTMs [26], normalization layers [27,33], or improved element-wise nonlinearities [31].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, methodology): “In fact, there are now a number of works that provide alternatives to BatchNorm [1, 3, 13, 31], but none of them seem to bring us any closer to understanding this issue.”
Understanding and Improving Layer Normalization (2019, background): “In addition, there are several studies exploring better activation functions [Klambauer et al., 2017] or initialization methods [Zhang et al., 2019] to avoid the dependency on normalization layers.”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “, 2016), SELU or RELU activation functions (Krizhevsky et al., 2012; Klambauer et al., 2017), LSTMs or GRUs (Hochreiter & Schmidhuber, 1997; Cho et al.”
Argumentative Relation Classification as Plausibility Ranking (2019, background): “Lastly, a selu-function (Klambauer et al., 2017) produces the desired plausibility-score: This score, computed once for each of the two competing reconstructions, allows a comparison with respect to their (predicted) plausibility.”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “FM2 is more subtle and does not seem to have been widely studied (see [16] for a notable exception).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “Another surprising connection is to the analysis of self-normalizing neural networks (Klambauer et al., 2017).”
YOLOv4: Optimal Speed and Accuracy of Object Detection (2020, methodology): “For improving the object detection training, a CNN usually uses the following: • Activations: ReLU, leaky-ReLU, parametric-ReLU, ReLU6, SELU, Swish, or Mish • Bounding box regression loss: MSE, IoU, GIoU, CIoU, DIoU • Data augmentation: CutOut, MixUp, CutMix • Regularization method: DropOut, DropPat”
A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images (2018, background): “SELUs is designed to make the forward neural network(FNN) to have self-normalizing capability[30].”
Application of Generative Autoencoder in De Novo Molecular Design (2017, methodology): “Instead of using rectiﬁed linear units [24] SELU [23] was used as non-linear activation function allowing a faster convergence of the model.”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “One the other hand, one of the insights from neural architecture search for DP models performed by Cheng et al. (2021) is that SELU activation (Klambauer et al., 2017) is more suitable than tempered sigmoids for DP Training.”