Efficient Distributed SGD with Variance Reduction

By Soham De, Tom Goldstein. 2015

43 citations recorded by the source corpus.

Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii) SGD scales poorly in distributed settings, typically experiencing rapidly decreasing marginal benefits as the number of workers increases. In this paper, we propose a highly parallel method, CentralVR, that uses error corrections to reduce the variance of SGD gradient updates, and scales linearly with the number of worker nodes. CentralVR enjoys low iteration complexity, provably linear convergence rates, and exhibits linear performance gains up to hundreds of cores for massive datasets. We compare CentralVR to state-of-the-art parallel stochastic optimization methods on a variety of models and datasets, and find that our proposed methods exhibit stronger scaling than other SGD variants.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the works for instance are Reddi et al. (2015); Zhao and Li (2016); De and Goldstein (2015); Lee et al. (2015).”
Collaborative Deep Learning in Fixed Topology Networks (2017, methodology): “A method called CentralVR [16] was proposed for reducing the variance and conducting parallel execution with linear convergence rate.”
Big Batch SGD: Automated Inference using Adaptive Batch Sizes (2016, background): “…enjoy a provably faster convergence rate than SGD and have been shown to outperform SGD on convex problems (Defazio et al., 2014a; Johnson and Zhang, 2013; Schmidt et al., 2013; Defazio et al., 2014b), as well as in parallel (Reddi et al., 2015) and distributed settings (De and Goldstein, 2016).”
Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks (2019, background): “In distributed SAGA, each worker maintains a table of stochastic gradients for all of its local data samples [28], [29].”
A Stochastic Large-scale Machine Learning Algorithm for Distributed Features and Observations (2018, methodology): “An alternative method which introduces the concept of variance reduced is CentralVR [6], where the master node not only needs to spread parameters but also the full gradient after every certain number of iterations, and each worker node would involve the full gradient as a corrector when computing t”