SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

By Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael G. Rabbat. 2019

224 citations recorded by the source corpus.

Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decouple communications among workers. Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates. Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (SlowMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm. Experiments on image classification and machine translation tasks demonstrate that SlowMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SlowMo runtime is on par with that of the base optimizer. We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses. Since BMUF can be expressed through the SlowMo framework, our results also correspond to the first theoretical convergence guarantees for BMUF.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “[402] proposed another momentum scheme called SlowMo, which can significantly improve the optimization and generalization performance of local SGD without sacrificing throughput.”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “The learning rate for the server optimizer is fixed to 1 as in [35, 32].”
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “[25] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat.”
Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “We note that in recent work more sophisticated local stochastic gradient methods have been considered, for example with momentum (Yu et al., 2019a; Wang et al., 2019), with quantization (Reisizadeh et al., 2019; Basu et al., 2019), with adaptive stepsizes (Xie et al., 2019) and with various…”
FetchSGD: Communication-Efficient Federated Learning with Sketching (2020, unknown): “Methods that use momentum in conjunction with local SGD have been proposed (Wang et al., 2019b) but the theoretical guarantees of using such methods in the federated setting are unclear.”
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning (2021, methodology): “This assumption is also used in other works for FL under non-i.i.d. datasets (Reddi et al., 2020; Yu et al., 2019b; Wang et al., 2019b) as well as in decentralized optimization (Kairouz et al., 2019).”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “8 Too many local steps may degrade the quality of the final model, and how to tune s is still an open research area [106, 105, 102, 64, 108, 50].”
Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD (2020, methodology): “Inspired by the distributed momentum scheme proposed in [18], Overlap-LocalSGD adopts a two-layer momentum structure.”
Federated Learning under Arbitrary Communication Patterns (2021, methodology): “…on synchronous local SGD with non-identical data distribution across clients wherein all clients communicate their local parameter to the server every fxed Δ rounds (Yu et al., 2019b; Haddadpour & Mahdavi, 2019; Khaled et al., 2019b;a; Wang et al., 2019c; Basu et al., 2019; Li et al., 2019b).”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “Substantial efforts have been made to address client heterogeneity in FL (Liang et al., 2019; Li et al., 2020b,a; Wang et al., 2020a; Zhang et al., 2021; Haddadpour et al., 2021; Yuan and Li, 2022; Alghunaim, 2023; Cheng et al., 2024), and develop other FL protocols involving variance reduction…”
Accelerating Federated Learning via Momentum Gradient Descent (2019, methodology): “In [26], momentum is applied to the update at each aggregation rounds for improving both optimization and generalization.”