AdaDelay: Delay Adaptive Distributed Stochastic Convex Optimization

By Suvrit Sra, Adams Wei Yu, Mu Li, Alex Smola. 2015

31 citations recorded by the source corpus.

We study distributed stochastic convex optimization under the delayed gradient model where the server nodes perform parameter updates, while the worker nodes compute stochastic gradients. We discuss, analyze, and experiment with a setup motivated by the behavior of real-world distributed computation networks, where the machines are differently slow at different time. Therefore, we allow the parameter updates to be sensitive to the actual delays experienced, rather than to worst-case bounds on the maximum delay. This sensitivity leads to larger stepsizes, that can help gain rapid initial convergence without having to wait too long for slower machines, while maintaining the same asymptotic complexity. We obtain encouraging improvements to overall convergence for distributed experiments on real datasets with up to billions of examples and features.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Distributed Learning over Unreliable Networks (2018, background): “Recent works indicate that many distributed learning is delay-tolerant under an asynchronous setting (Zhou et al., 2018; Lian et al., 2015; Sra et al., 2015; Leblond et al., 2016).”
A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates (2018, result): “We note that there are several other works which study a similar setting (such as [22]), but do not result in bounds which improve on the above.”
Asynchronous Incremental Stochastic Dual Descent Algorithm for Network Resource Allocation (2017, background): “It is worth pointing out that there is a subtle difference between the definition of the delayed stochastic gradient considered here, and those considered in [22], [39], [40].”
Asynchronous Stochastic Gradient Descent with Delay Compensation (2016, background): “Researchers have tried to improve ASGD from different perspectives (Ho et al., 2013; McMahan & Streeter, 2014; Zhang et al., 2015; Sra et al., 2015; Mitliagkas et al., 2016), however, to the best of our knowledge, there is still no solution that can compensate the delayed gradient while keeping the…”
Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design (2016, result): “This is joint work with Suvrit Sra, Adams Yu and Alex Smola, and the results were published in AISTATS’16 [16].”