rTop-k: A Statistical Estimation Approach to Distributed SGD
By Leighton Pate Barnes, Huseyin A. Inan, Berivan Isik, Ayfer Özgür. 2020
71 citations recorded by the source corpus.
The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivated by this observation, there has been significant recent interest in techniques that reduce the communication cost of distributed Stochastic Gradient Descent (SGD), with gradient sparsification techniques such as top-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> and random-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> shown to be particularly effective. The same observation has also motivated a separate line of work in distributed statistical estimation theory focusing on the impact of communication constraints on the estimation efficiency of different statistical models. The primary goal of this paper is to connect these two research lines and demonstrate how statistical estimation models and their analysis can lead to new insights in the design of communication-efficient training techniques. We propose a simple statistical estimation model for the stochastic gradients which captures the sparsity and skewness of their distribution. The statistically optimal communication scheme arising from the analysis of this model leads to a new sparsification technique for SGD, which concatenates random-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> and top-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>, considered separately in the prior literature. We show through extensive experiments on both image and language domains with CIFAR-10, ImageNet, and Penn Treebank datasets that the concatenated application of these two sparsification methods consistently and significantly outperforms either method applied alone.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ayfer Özgür
- Wei-Ning Chen
- Peter Kairouz
- Vitaly Feldman
- Kunal Talwar
- Pramod Viswanath
- Jayadev Acharya
- Sewoong Oh
In Computer Vision and Pattern Recognition
In Optimization & Training
In ML Systems & Hardware
Citations in the corpus
- Breaking the Communication-Privacy-Accuracy Trilemma (2020, background): “Previous works in the mean estimation problem [8], [12], [20], [21], [22], [23] mainly focus on reducing communication cost, for instance, by random rotation [12] and sparsification [20], [21], [24], [25].”