Asynchronous parallel stochastic gradient descent: a numeric core for scalable distributed machine learning algorithms
By Janis Keuper, Franz-Josef Pfreundt. 2015
38 citations recorded by the source corpus.
The implementation of a vast majority of machine learning (ML) algorithms boils down to solving a numerical optimization problem. In this context, Stochastic Gradient Descent (SGD) methods have long proven to provide good results, both in terms of convergence and accuracy. Recently, several parallelization approaches have been proposed in order to scale SGD to solve very large ML problems. At their core, most of these approaches are following a MapReduce scheme. This paper presents a novel parallel updating algorithm for SGD, which utilizes the asynchronous single-sided communication paradigm. Compared to existing methods, Asynchronous Parallel Stochastic Gradient Descent (ASGD) provides faster convergence, at linear scalability and stable accuracy.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kumar Kshitij Patel
- Aymeric Dieuleveut
- Tomasz Grel
- Robert Adamski
- Kamil Kaczmarek
- Igor Adamski
- Igor Adamski
- Henryk Michalewski
In Computational Mechanics
Citations in the corpus
- Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “…order are Langford et al. (2009); Niu et al. (2011); Agarwal and Duchi (2011); Paine et al. (2013); Li et al. (2014b); Zhang et al. (2014); Keuper and Pfreundt (2015); De and Goldstein (2015); Feyzmahdavian et al. (2015); Lian et al. (2015); Mania et al. (2015); Zhao and Li (2015); Duchi…”
- Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, background): “For a survey of the various asynchronous gradient descent procedures we refer the reader to [9] and [16].”