The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study

By Daniel S. Park, Jascha Narain Sohl-Dickstein, Quoc V. Le, Samuel L. Smith. 2019

61 citations recorded by the source corpus.

We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, and network width. We find that the optimal SGD hyper-parameters are determined by a "normalized noise scale," which is a function of the batch size, learning rate, and initialization conditions. In the absence of batch normalization, the optimal normalized noise scale is directly proportional to width. Wider networks, with their higher optimal noise scale, also achieve higher test accuracy. These observations hold for MLPs, ConvNets, and ResNets, and for two different parameterization schemes ("Standard" and "NTK"). We observe a similar trend with batch normalization for ResNets. Surprisingly, since the largest stable learning rate is bounded, the largest batch size consistent with the optimal normalized noise scale decreases as the width increases.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…is that noise from stochastic gradient descent leads to ﬂat minima, and that ﬂat minima generalize better than sharp minima (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Smith & Le, 2018; Jiang et al., 2020; Park et al., 2019) (though see Dinh et al. (2017) for discussion of some caveats).”
On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “Furthermore, when considering unlimited compute budgets, we find evidence of an “optimal temperature” that promotes generalization (Smith & Le, 2017; Park et al., 2019).”
On the infinite width limit of neural networks with a standard parameterization (2020, methodology): “• As the naive standard parameterization is extended to large widths, the largest stable learning rate scales like 1 width [14, Theorem 7]; [25, §H].”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “[26] studied how learning rate (and batch size) should scale with width for MLPs and CNNs trained with SGD in NTK or standard parametrizations.”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, background): “However, when the batch size is ﬁxed, this ratio is proportional to the product of the learning rate and the augmentation multiplicity, which we deﬁne for clarity as the ‘temperature’ [Mandt et al., 2017, Smith and Le, 2018, Park et al., 2019].”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “Note that Park et al. (2019) show that, as long as the network size is not too big, controlling for “optimality of step size” and “keeping step size constant” with increasing network width actually correspond to the same thing, as long as the networks are not too big, which ﬁts well with the fact…”
On the Intrinsic Privacy of Stochastic Gradient Descent (2019, background): “The properties of SGD have since become an object of study [6], [39], [45], with particular focus on the generalisation performance of the solutions it ﬁnds [14], [22], [25].”
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths (2020, background): “It breaks the independence between training samples in a minibatch, which makes it hard to apply in certain models (Girshick, 2015), and also contradicts the assumptions of most theoretical models of optimization (Mandt et al., 2017; Park et al., 2019).”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, methodology): “…learning dynamics in this limit also simpliﬁes [Jacot et al., 2018, Du et al., 2019, Allen-Zhu et al., 2018, Lee et al., 2019, Arora et al., 2019, Park et al., 2019] and is entirely described by the neural tangent kernel (NTK) [Jacot et al., 2018] deﬁned as: where x 1 , x 2 are two inputs and ∇…”