Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models

By Gideon S. Mann, Ryan T. McDonald, Mehryar Mohri, Nathan Silberman, Dan Walker. 2009

311 citations recorded by the source corpus.

Subfields: Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Multiple-Source Adaptation with Domain Classifiers (2020, neutral): “From Theorem 2 of McDonald et al. [2009], for any δ > 0, with probability at least 1− δ, ‖ŵ − w∗‖ ≤ R ξ √ m/2 (1 + √ log 1/δ).”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Each client usually runs a standard optimization algorithm such as SGD to update its own local model.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, methodology): “In this approach, the P -independent workers compute several steps of stochastic gradient descent, and a unique communication step is used to average the different models (Mcdonald et al., 2009; McDonald et al., 2010; Zinkevich et al., 2010).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “…system resources (computation vs. communication), local SGD (a.k.a. local-update SGD, parallel SGD, or federated averaging) has recently attracted increased research interest (Mcdonald et al., 2009; Zinkevich et al., 2010; McDonald et al., 2010; Zhang et al., 2014; 2016; McMahan et al., 2017).”
Distributed learning of deep neural network over multiple agents (2018, methodology): “Interesting techniques include distributed gradient optimization[7, 8], online learning with delayed updates[9] and hashing and simpliﬁcation of kernels[10].”
Local SGD Converges Fast and Communicates Little (2018, background): “An extreme instance of this concept is one-shot SGD [21, 53] where the local sequences are only exchanged once, after the local runs have converged.”
CoCoA: A General Framework for Communication-Efficient Distributed Optimization (2016, methodology): “At the other extreme, there are distributed methods that use only a single round of communication, such as Mann et al. (2009); Zinkevich et al. (2010); Zhang et al. (2013); McWilliams et al. (2014); and Heinze et al. (2016).”
Adding vs. Averaging in Distributed Primal-Dual Optimization (2015, methodology): “At the other extreme, there are distributed methods using only a single round of communication, such as (Zhang et al., 2013; Zinkevich et al., 2010; Mann et al., 2009; McWilliams et al., 2014).”
Communication-Efficient Distributed Dual Coordinate Ascent (2014, background): “We see that thoughβK has a larger impact on the smaller batch size, it is still not enough to improve the mini-batch algorithms beyond what is achieved by COCOA and local-SGD.”
Distributed optimization with arbitrary local solvers (2015, methodology): “At the other extreme, there are distributed methods using only a single round of communication, such as [24, 36, 38, 79, 82].”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, background): “Second, it is shown that model averaging for convex models is guaranteed to converge [17, 18].”
Efficient Decentralized Deep Learning by Dynamic Model Averaging (2018, neutral): “We investigate synchronization operators that aggregate models by computing their average [25, 24, 33, 45, 44], i.”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “…the computation to communication ratio, such as increased mini-batch sizes (Goyal et al., 2017), defining local problems for each worker (Shamir et al., 2014; Reddi et al., 2016) or reducing the communication frequency (McDonald et al., 2009; Zinkevich et al., 2010; You et al., 2017; Stich, 2018).”
Parallel Coordinate Descent for L1-Regularized Loss Minimization (2011, background): “Recent work analyzes parallel stochastic gradient descent for multicore (Langford et al., 2009b) and distributed settings (Mann et al., 2009; Zinkevich et al., 2010).”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “At the other extreme, there are distributed methods using only a single round of communication, such as [198, 203, 114, 76, 75].”
Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification (2016, background): “Parameter mixing/Model averaging (Mann et al., 2009) guarantees linear parallelization speedups on the variance but do not improve the bias.”