Deep Mean Field Theory: Layerwise Variance and Width Variation as Methods to Control Gradient Explosion

By Greg Yang, Sam Schoenholz. 2018

24 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “This evolution is referred to as (forward and backward) signal propagation in the literature (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018; Pennington et al., 2017).”
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks (2018, methodology): “…work has used mean ﬁeld theory to build a theoretical understanding of neural networks with random parameters (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
How to Start Training: The Effect of Initialization and Architecture (2018, methodology): “The articles [32, 33], in contrast, use mean ﬁeld theory to derive the dependence of both forward and backward dynamics in a randomly initialized ResNet on the residual module weights.”
A Mean Field Theory of Batch Normalization (2019, background): “Over the past year, this gap has closed significantly and theory for networks with skip connections (Yang & Schoenholz, 2017; 2018), convolutional networks (Xiao et al., 2018), and gated ∗Please see https://arxiv.org/abs/1902.08129 for the full and most current version of this paper ar X iv :1 90 2.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “The line of work examining signal propagation in random deep networks (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018) is related to the construction of the GPs we consider.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “…theory to explore questions of trainability and expressivity in fully-connected networks and residual networks (Poole et al., 2016; Schoen-holz et al., 2017; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Karakida et al., 2018; Hayou et al., 2018; Hanin & Rolnick, 2018; Yang & Schoenholz, 2018).”
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, background): “…have been developed to control training dynamics and train models stably (Glorot & Bengio, 2010; Yang & Schoenholz, 2017; Schoenholz et al., 2017; Yang & Schoenholz, 2018; Zhang et al., 2019; Bachlechner et al., 2020; Huang et al., 2020; Liu et al., 2020; Li et al., 2022). µP is the ﬁrst…”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, methodology): “[23] studies the effect of initialization in deep neural networks [46, 50, 63, 62, 21, 9, 64, 45], for example, what is the best initialization scheme to avoid gradient vanishing? These works apply the same calculations of covariances as we do for calculating Σ here, though in a much more restricted”
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers (2019, result): “This is consistent with work which suggests that theoretically-grounded initialization schemes can enable the training of extremely deep networks with hundreds of layers [13, 27, 29, 33, 35].”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “Many previous works proposed different initialization or parametrizations with favorable properties, such as better stability for training deep neural networks [5, 10, 13, 21, 30, 47, 48, 51].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Closely related to the discussion of CK and NTK is the signal propagation literature, which tries to understand how to prevent pathological behaviors in randomly initialized neural networks when they are deep [48, 52, 71, 70, 23, 24, 14, 72, 44, 25, 47].”