Adjusting for Dropout Variance in Batch Normalization and Weight Initialization

17 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Gaussian Error Linear Units (GELUs) (2016, background): “Weights are initialized with unit norm rows, as this has positive impact on each nonlinearity’s performance (Hendrycks & Gimpel, 2016; Mishkin & Matas, 2016; Saxe et al., 2014).”
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, methodology): “Note that treating error/abnormal classes as positive classes does not change the AU- 1We consider adversarial example detection techniques in a separate work (Hendrycks & Gimpel, 2016a).”
Regularization for Deep Learning: A Taxonomy (2017, background): “• Initialization without pre-training – Random weight initialization (Rumelhart et al., 1986, p. 330; Glorot and Bengio, 2010; He et al., 2015; Hendrycks and Gimpel, 2016) – Orthogonal weight matrices (Saxe et al., 2013) – Data-dependent weight initialization (Krähenbühl et al., 2015) •…”
Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift (2018, background): “We discover the key to understand the disharmony between Dropout and BN is the inconsistent behaviors of neural variance [12] during the switch of networks’ state.”