The Effect of Network Width on the Performance of Large-batch Training

By Lingjiao Chen, Hongyi Wang, Jinman Zhao, Dimitris Papailiopoulos, Paraschos Koutris. 2018

24 citations recorded by the source corpus.

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and generalization performance of SGD. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that--for a fixed number of parameters--wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Torsten Hoefler
2. Jaehoon Lee
3. Jascha Narain Sohl-Dickstein
4. Dan Alistarh
5. Greg Yang
6. Percy Liang
7. Martin Jaggi
8. George E. Dahl

In Management Science and Operations Research

1. Joseph M. Antognini
2. Jascha Narain Sohl-Dickstein
3. Torsten Hoefler
4. Christopher J. Shallue
5. Dan Alistarh
6. Roy Frostig
7. George E. Dahl
8. Jaehoon Lee

In Optimization & Training

1. Torsten Hoefler
2. Jaehoon Lee
3. Christopher J. Shallue
4. Jascha Narain Sohl-Dickstein
5. Joseph M. Antognini
6. Dan Alistarh
7. Roy Frostig
8. George E. Dahl

Citations in the corpus

• Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “This contrasts with the findings of Chen et al. (2018), although they changed width and depth simultaneously while keeping the number of parameters fixed.”
• SparCML: High-Performance Sparse Communication for Machine Learning (2018, background): “It has been empirically found that shallow variants of wide models can achieve similar or better levels of accuracy as considerably deeper architectures [60], and that they are less sensitive to hyperparametrization, and in particular to large-batch training [12].”