Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go?

67 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Furthermore, [50] indicates that even polynomially growing delays can be tolerated by utilizing a quasilinear step-size sequence, but without achieving linear speedup.”
Zeno++: Robust Fully Asynchronous SGD (2019, background): “Recent years have witnessed an increasing number of largescale machine learning algorithms, including asynchronous SGD (Zinkevich et al., 2009; Lian et al., 2018; Zheng et al., 2017; Zhou et al., 2018).”
Distributed Learning over Unreliable Networks (2018, background): “Recent works indicate that many distributed learning is delay-tolerant under an asynchronous setting (Zhou et al., 2018; Lian et al., 2015; Sra et al., 2015; Leblond et al., 2016).”
Local SGD Converges Fast and Communicates Little (2018, background): “Asynchronous SGD converges also for larger delays (see also [52]) but without linear speedup, a similar statement holds for local SGD (see discussion in Section 3).”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, unknown): “org/ the ASP [129].”