Don't Use Large Mini-Batches, Use Local SGD

By Tao Lin, Sebastian U. Stich, Martin Jaggi. 2018

470 citations recorded by the source corpus.

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Assuming an epoch budget when comparing solution quality between batch sizes (Masters and Luschi, 2018; Goyal et al., 2017; Lin et al., 2018; Devarakonda et al., 2017), in eﬀect, limits an investigation to the perfect scaling region of the steps to result vs batch size curve (see Figure 1).”
Advances and Open Problems in Federated Learning (2019, background): “The convergence analysis of local-update SGD methods is an active current area of research [370, 271, 428, 399, 334, 318, 233].”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “, 2018a; 2019; Stich & Karimireddy, 2019)—but such orthogonal techniques could be added on top of our scheme—and instead only focus on local updates steps which are often efficient in practice but challenging to handle in the theoretical analysis (McMahan et al., 2017; Stich, 2019b; Yu et al., 2019;”
Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “This includes quantization and compression ideas (Reisizadeh et al., 2019; Dai et al., 2019) as well as performing multiple local updates before communicating with the master (Stich, 2018; Lin et al., 2018; Wang and Joshi, 2018).”
Is Local SGD Better than Minibatch SGD? (2020, methodology): “A string of recent papers have attempted to analyze local SGD for convex objectives, (e.g. Stich, 2018; Stich and Karimireddy, 2019; Khaled et al., 2019; Dieuleveut and Patel, 2019).”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “In fact, there is previous work (Lin et al., 2018) showing that local SGD potentially generalizes better than the fully synchronous SGD, which makes our results not surprising.”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Recently, [45] and [20] have shown empirically that more frequent communication in the beginning leads to improved performance over fixed communication period.”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, methodology): “Beyond them, a simple but powerful method is to reduce the communication frequency by allowing more local updates [58, 39, 22, 47, 44], which we focus on in this paper.”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “Recent rigorous theoretical analysis (Stich, 2018; Wang and Joshi, 2018; Yu et al., 2018; Lin et al., 2018) shows that FedAvg is a special case of averaging periodic SGD (also called local SGD) which allows nodes to perform local updates and infrequent synchronization between them to communicate…”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “Lin et al. (2018) apply Local SGD for distributed training of deep neural networks and propose postlocal SGD, which starts by running ALLREDUCE-SGD for some epochs before switching to Local SGD, to improve generalization at the cost of additional communication.”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, background): “Stich [2018] considered a master-worker topology and provides theoretical analysis for the convergence of local-SGD method.”
Gradient Descent with Compressed Iterates (2019, background): “Another line of work focused on local stochastic gradient methods that communicate only intermittently and average models, such as Local SGD [19, 27, 30, 31] and Federated Averaging [21], and combinations of such methods and update quantization (where the sum of gradients over an epoch is quantized)”
On the Convergence of FedAvg on Non-IID Data (2019, background): “7 [14] Tao Lin, Sebastian U Stich, and Martin Jaggi.”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “Such a method is also faithful to the common practice “model averaging with momentum” used by practitioners for training deep neural networks in (Chen & Huo, 2016; McMahan et al., 2017; Su et al., 2018; Lin et al., 2018).”