Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
By Cong Xie, Oluwasanmi Koyejo, Indranil Gupta, Haibin Lin. 2019
46 citations recorded by the source corpus.
Recent years have witnessed the growth of large-scale distributed machine learning algorithms -- specifically designed to accelerate model training by distributing computation across multiple machines. When scaling distributed training in this way, the communication overhead is often the bottleneck. In this paper, we study the local distributed Stochastic Gradient Descent~(SGD) algorithm, which reduces the communication overhead by decreasing the frequency of synchronization. While SGD with adaptive learning rates is a widely adopted strategy for training neural networks, it remains unknown how to implement adaptive learning rates in local SGD. To this end, we propose a novel SGD variant with reduced communication and adaptive learning rates, with provable convergence. Empirical results show that the proposed algorithm has fast convergence and efficiently reduces the communication overhead.
Subfields: Artificial Intelligence, Optimization & Training, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ahmed Khaled
- Peter Richtárik
- Konstantin Mishchenko
- Nicolas Loizou
- Filip Hanzely
- Sebastian U. Stich
- Jakub Konecný
- Dmitry Kovalev
In Optimization & Training
- Ahmed Khaled
- Peter Richtárik
- Konstantin Mishchenko
- Nicolas Loizou
- Filip Hanzely
- Jakub Konecný
- Sebastian U. Stich
- Martin Jaggi
In ML Systems & Hardware
- Konstantin Mishchenko
- Ahmed Khaled
- Peter Richtárik
- Martin Takác
- Jakub Konecný
- Nicolas Loizou
- Nati Srebro
- Stephen J. Wright
Citations in the corpus
- Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “…considered, for example with momentum (Yu et al., 2019a; Wang et al., 2019), with quantization (Reisizadeh et al., 2019; Basu et al., 2019), with adaptive stepsizes (Xie et al., 2019) and with various variance-reduction methods (Liang et al., 2019; Sharma et al., 2019; Karimireddy et al., 2019).”