Generalizing and Improving Weight Initialization

By Dan Hendrycks, Kevin Gimpel. 2016

9 citations recorded by the source corpus.

We propose a new weight initialization suited for arbitrary nonlinearities by generalizing previous weight initializations. The initialization corrects for the influence of dropout rates and an arbitrary nonlinearity's influence on variance through simple corrective scalars. Consequently, this initialization does not require computing mini-batch statistics nor weight pre-initialization. This simple method enables improved accuracy over previous initializations, and it allows for training highly regularized neural networks where previous initializations lead to poor convergence.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Dan Hendrycks
2. Kevin Gimpel
3. Daniel Cremers
4. Vladimir Golkov
5. Jan Kukacka
6. Daniel Klein
7. Greg Durrett

Citations in the corpus

• Gaussian Error Linear Units (GELUs) (2016, methodology): “. We use the Adam optimizer and its suggested learning rate of 0:001 [6]. The weights are uniformly initialized on the unit hypersphere, as this has positive impact on each nonlinearity’s performance [4, 9, 11]. Last, note that we perform this task with no dropout and a dropout rate of 0:5. Figure 2”
• Regularization for Deep Learning: A Taxonomy (2017, background): “∙ Initialization without pre-training – Random weight initialization (Rumelhart et al., 1986, p. 330; Glorot and Bengio, 2010; He et al., 2015; Hendrycks and Gimpel, 2016) – Orthogonal weight matrices (Saxe et al.”