A Parallel SGD method with Strong Convergence

By Dhruv Kumar Mahajan, S. Sathiya Keerthi, S. Sundararajan, L. Eon Bottou. 2013

16 citations recorded by the source corpus.

This paper proposes a novel parallel stochastic gradient descent (SGD) method that is obtained by applying parallel sets of SGD iterations (each set operating on one node using the data residing in it) for finding the direction in each iteration of a batch descent method. The method has strong convergence properties. Experiments on datasets with high dimensional feature spaces show the value of this method.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Communication-Efficient Distributed Optimization using an Approximate Newton-type Method (2013, methodology): “A similar dependence occurs with other recently-proposed algorithms for distributed optimization (e.g. Yang, 2013; Mahajan et al., 2013; Dekel et al., 2012; Cotter et al., 2011; Duchi et al., 2012).”
Communication Complexity of Distributed Convex Learning and Optimization (2015, background): “As datasets continue to increase in size, and parallel computing platforms becoming more and more common (from multiple cores on a single CPU to large-scale and geographically distributed computing grids), distributed learning and optimization methods have been the focus of much research in recent y”
GIANT: Globally Improved Approximate Newton Method for Distributed Optimization (2017, methodology): “Stochastic ﬁrst-order methods 1 , including distributed variants of stochastic gradient descent (SGD) (Mahajan et al., 2013, Recht et al., 2011, Zinkevich et al., 2010), accelerated SGD (Shamir and Srebro, 2014), variance reduction SGD (Lee et al., 2015, Reddi et al., 2015), substantially reduce…”
Efficient mini-batch training for stochastic optimization (2014, background): “In addition, [19, 13] argued to reduce the stochastic variance via gradients computed on the whole dataset.”
An efficient distributed learning algorithm based on effective local functional approximations (2013, methodology): “This contribution is covered in detail in Mahajan et al. (2013b); we give a summarized view in Subsection 3.5.”
Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design (2016, unknown): “In addition, [72, 93] argued to reduce the stochastic variance via gradients computed on the whole dataset.”
A Functional Approximation Based Distributed Learning Algorithm (2013, methodology): “One could also use methods with glrc in the expectation sense (in which case, convergence in Theorem 4 should also be interpreted in some probabilistic sense; see our related work [15] for details).”