On the Selection of Initialization and Activation Function for Deep Neural Networks

By Soufiane Hayou, Arnaud Doucet, Judith Rousseau. 2018

77 citations recorded by the source corpus.

The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `edge of chaos' can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, $\phi_{swish}(x) = x \cdot \text{sigmoid}(x)$, used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of $\phi_{swish}$ observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “…work has used mean ﬁeld theory to build a theoretical understanding of neural networks with random parameters (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, background): “…mean ﬁeld theory and initialization (Cho & Saul, 2009; Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2016; Yang & Schoenholz, 2017; Xiao et al., 2018; Li & Nguyen, 2019; Pretorius et al., 2018; Hayou et al., 2018; Karakida et al., 2018; Blumenfeld et al., 2019; Hayou et al., 2019).”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “…theory to explore questions of trainability and expressivity in fully-connected networks and residual networks (Poole et al., 2016; Schoen-holz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Tensor Programs III: Neural Matrix Laws (2020, background): “[16] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “[25] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau.”
Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, background): “[29] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau.”
Model complexity of deep learning: a survey (2021, background): “Such a metric also helps to verify the effectiveness of new improvements of optimization algorithms [39].”
A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate (2021, background): “Some suggestions upon the choice of activation function can be referred in [54].”
Disentangling trainability and generalization in deep learning (2019, background): “, 2018), a range of activation functions (Hayou et al., 2018), batch normalization (Yang et al.”
Appendix for: How Good is the Bayes Posterior in Deep Neural Networks Really? (2020, unknown): “Glorot & Bengio (2010) and He et al. (2015) developed efficient random initialization schemes for deep neural networks and a more formal analysis of information flow in random functions defined by neural networks is given by Schoenholz et al. (2017) and Hayou et al. (2018).”
Mish: A Self Regularized Non-Monotonic Neural Activation Function (2019, unknown): “These two are: Rectified Linear Unit (ReLU) [1,2,3] which can be defined by 𝑓(𝑥) = max⁡(0, 𝑥) and Swish [4,5] which can be defined as: 𝑓(𝑥) = 𝑥 ⋅ 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝑥) .”
Deep Learning vs. Traditional Computer Vision (2019, background): “Depending on the nature of data and classification tasks, these activation functions are selected accordingly [11].”