Delay-Tolerant Algorithms for Asynchronous Distributed Online Learning

By H. Brendan McMahan, Matthew J. Streeter. 2014

133 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization (2018, background): “[16] Brendan McMahan and Matthew Streeter.”
An overview of gradient descent optimization algorithms (2016, background): “mmunicate with each other e.g. by sharing weights or updates, their parameters are continuously at risk of diverging, hindering convergence. 5.3 Delay-tolerant Algorithms for SGD McMahan and Streeter [12] extend AdaGrad to the parallel setting by developing delay-tolerant algorithms that not only ad”
An asynchronous mini-batch algorithm for regularized stochastic optimization (2015, neutral): “, [21]–[23].”
On Nonconvex Decentralized Gradient Descent (2016, background): “Furthermore, although DGD is slower than DADMM, DLM and EXTRA on convex problems, DGD is simpler and thus easier to extend to a variety of settings such as [23], [38], [47], [64], where online processing and delay tolerance are considered.”
Asynchronous Decentralized Online Learning (2021, methodology): “This term is in line with the delay effect term in the asynchronous centralized online setting [18].”
AdaDelay: Delay Adaptive Distributed Stochastic Optimization (2016, methodology): “Our experiments also confirm [12]’s claims that their best learning rate is insensitive to maximum delays.”
Toward Understanding the Impact of Staleness in Distributed Machine Learning (2018, background): “…Langford et al. (2009); Lian et al. (2015); Recht et al. (2011) empirically and theoretically show that many algorithms scale effectively under non-synchronous settings, but McMahan & Streeter (2014); Mitliagkas et al. (2016); Hadjis et al. (2016) demonstrate signiﬁcant penalties from asynchrony.”
Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design (2016, unknown): “Of particular relevance to our work is the recent work on delay adaptive gradient scaling in an AdaGrad like framework [98], which claims substantial improvements under specialized settings over [50], a work that exploits data sparsity in a distributed asynchronous setting.”
Asynchronous Stochastic Gradient Descent with Delay Compensation (2016, background): “Researchers have tried to improve ASGD from different perspectives (Ho et al., 2013; McMahan & Streeter, 2014; Zhang et al., 2015; Sra et al., 2015; Mitliagkas et al., 2016), however, to the best of our knowledge, there is still no solution that can compensate the delayed gradient while keeping the…”