CSER: Communication-efficient SGD with Error Reset
By Cong Xie, Shuai Zheng, Oluwasanmi Koyejo, Indranil Gupta, Mu Li, Haibin Lin. 2020
48 citations recorded by the source corpus.
The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: Communication-efficient SGD with Error Reset, or CSER. The key idea in CSER is first a new technique called "error reset" that adapts arbitrary compressors for SGD, producing bifurcated local models with periodic reset of resulting local residual errors. Second we introduce partial synchronization for both the gradients and the models, leveraging advantages from them. We prove the convergence of CSER for smooth non-convex problems. Empirical results show that when combined with highly aggressive compressors, the CSER algorithms: i) cause no loss of accuracy, and ii) accelerate the training by nearly $10\times$ for CIFAR-100, and by $4.5\times$ for ImageNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cong Xie
- Oluwasanmi Koyejo
- Indranil Gupta
- Haibin Lin
- Lifeng Lai
- Xinyang Cao
- Rachid Guerraoui
- El-Mahdi El-Mhamdi
In Computer Vision and Pattern Recognition
Citations in the corpus
- Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, background): “There are other approaches to reduce the communication overhead of distributed SGD, such as quantization [3, 4, 12, 22, 27, 32, 42] and sparsification [2, 10, 26, 33].”
- 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (2021, unknown): “This method can be combined with decentralized training (Vogels et al., 2020), local SGD (Xie et al., 2020), accelerated algorithms (Gorbunov et al., 2020).”