Scaling up machine learning: parallel and distributed approaches

By Ron Bekkerman, Mikhail Bilenko, J. Langford. 2011

451 citations recorded by the source corpus.

Subfields: Artificial Intelligence, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In ML Systems & Hardware

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “…research efforts are invested in speeding up DNNs at run-time on both generalpurpose (Vanhoucke et al., 2011; Gong et al., 2014; Romero et al., 2014; Han et al., 2015b) and specialized computer hardware (Farabet et al., 2011a,b; Pham et al., 2012; Chen et al., 2014a,b; Esser et al., 2015).”
Natural Compression for Distributed Deep Learning (2019, background): “[5] Ron Bekkerman, Mikhail Bilenko, and John Langford.”
Byzantine Stochastic Gradient Descent (2018, background): “This can be either because the data is naturally distributed—for instance, in applications such as federated learning , where data collection is crowdsourced [20]—or because data is partitioned across machines in order to parallelize computation, e.g. [4].”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, methodology): “There is an extremely rich area studying algorithms and systems for efﬁcient distributed large-scale learning, e.g. [5, 10, 1, 3, 34, 28, 9, 19].”
Deep learning with Elastic Averaging SGD (2014, background): “We however emphasize that to the best of our knowledge the problem of stochastic optimization in the parallel deep learning setting under the communication constraints was never addressed in the literature before (and thus as well a more general problem when the data is distributed).”
Communication-Efficient Distributed Optimization using an Approximate Newton-type Method (2013, methodology): “Since the cost of communication is very high in practice (Bekkerman et al., 2011), our goal is to develop methods which quickly optimize the empirical objective ˆ F ( · ) , using a minimal number of such iterations.”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, background): “Compared with local computation on any single node, the cost of communication between nodes is much higher both in terms of speed and energy consumption [6, 89], introducing new computational challenges, not only for optimization procedures.”
XGBoost: A Scalable Tree Boosting System (2016, methodology): “We summarize an approximate framework, which resembles the ideas proposed in past literatures [17, 2, 22], in Alg.”
Communication Complexity of Distributed Convex Learning and Optimization (2015, background): “As datasets continue to increase in size, and parallel computing platforms becoming more and more common (from multiple cores on a single CPU to large-scale and geographically distributed computing grids), distributed learning and optimization methods have been the focus of much research in recent y”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “…light of this, substantial research efforts are invested in speeding up DCNNs at both run-time and training-time, on both general-purpose (Vanhoucke et al., 2011; Gong et al., 2014; Han et al., 2015b) and specialized computer hardware (Farabet et al., 2011; Pham et al., 2012; Chen et al., 2014a;b).”
A reliable effective terascale linear learning system (2011, neutral): “In particular their throughput—measured as the input size divided by the wall clock running time—is smaller than the the I/O interface of a single machine for almost all parallel learning algorithms (Bekkerman et al., 2011, Part III, page 8).”
Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation (2013, background): “Some examples include: • Communication constraints in distributed learning: There has been much work in recent years on learning when the training data is distributed among several machines (with [14, 2, 28, 47, 25, 31, 9, 17, 38] being just a few examples).”
DSA: Decentralized Double Stochastic Averaging Gradient Algorithm (2015, background): “The formulation in (1) models a training set with a total of ∑N n=1 qn training samples that are distributed among the N agents for parallel processing conducive to the determination of the optimal classifier x̃∗ (Bekkerman et al. (2011); Tsianos et al.”
Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss (2015, neutral): “O C ] 1 Ja”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, background): “Compared with local computation on any single node, the cost of communication between nodes is much higher both in terms of speed and energy consumption [12, 165], introducing new computational challenges, not only for optimization procedures.”
Interpretable models from distributed data via merging of decision trees (2013, background): “There has been many excellent works on supervised learning in this domain [4], [5], [2].”