Distributed Training Strategies for the Structured Perceptron

By Ryan T. McDonald, Keith B. Hall, Gideon S. Mann. 2010

321 citations recorded by the source corpus.

Subfields: Federated & Distributed Learning, Algorithms & Complexity

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Federated & Distributed Learning

In Algorithms & Complexity

Citations in the corpus

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, methodology): “This motivates us to study the following key question: Can we reduce the number of communication rounds even more, and yet achieve linear speedup? In this paper, we give an affirmative answer to this question by providing a tighter analysis of local SGD via model averaging [43, 38, 51, 23, 45, 49].”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “The idea of reducing communication overhead by using ALLREDUCE to synchronize parameters after every τ > 0 optimizer steps has been considered at least since the work of McDonald et al. (2010), and has been more recently referred to as Local SGD in the literature.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, methodology): “In this approach, the P -independent workers compute several steps of stochastic gradient descent, and a unique communication step is used to average the different models (Mcdonald et al., 2009; McDonald et al., 2010; Zinkevich et al., 2010).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “…system resources (computation vs. communication), local SGD (a.k.a. local-update SGD, parallel SGD, or federated averaging) has recently attracted increased research interest (Mcdonald et al., 2009; Zinkevich et al., 2010; McDonald et al., 2010; Zhang et al., 2014; 2016; McMahan et al., 2017).”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “The method in (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010), referred to as one-shot averaging, uses only one single communication step at the end and is numerically shown to have good solution quality in many applications.”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Distributed stochastic gradient descent (SGD) is a basic building block of modern machine learning [25, 11, 9, 28, 1, 27, 5].”
Local SGD Converges Fast and Communicates Little (2018, background): “[22] for perceptron training (iterative parameter mixing), see also [7], [5, 46, 48] for the training of deep neural networks (model averaging) or in federated learning [23].”
Distributed Mean Estimation with Limited Communication (2016, neutral): “Recently, algorithms involving distributed mean estimation have been used extensively in training large-scale neural networks and other statistical models (McDonald et al., 2010; Povey et al., 2014; Dean et al., 2012; McMahan et al., 2016; Alistarh et al., 2016).”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, methodology): “[28] for the perceptron and Povey et al.”
Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling (2010, positive): “t are directly connected with small network latencies. A special case of our optimization problem within this computational model is the distributed perceptron, recently considered by McDonald et al. [MHM10]. 2.2 Standard dual averaging Our algorithm is based on a projected dual averaging algorithm ”
Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “While local SGD has been used among practitioners for a long time, see e.g. (Coppola, 2015; McDonald et al., 2010), its theoretical analysis has been limited until recently.”
A reliable effective terascale linear learning system (2011, methodology): “Some prior works (McDonald et al., 2010; Zinkevich et al., 2010) consider online learning with averaging and Duchi et al. (2010a) propose gossip-style message passing algorithms extending the existing literature on distributed convex optimization (Bertsekas and Tsitsiklis, 1989).”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “Stampede nodes run Linux 2.6.32 with batch services managed by the Simple Linux Utility for Resource Management (SLURM).”
Training a Natural Language Generator From Unaligned Data (2015, methodology): “Averaging weights and parallel training To speed up training using parallel processing, we use the iterative parameter mixing approach of McDonald et al. (2010), where training data are split into several parts and weight updates are averaged after each pass through the training data.”
On the Convergence of Local Descent Methods in Federated Learning (2019, methodology): “The references [24,40] introduce the one shot averaging, which can be considered as an extreme case of periodic averaging ( E = T ), and show empirically that one-shot averaging works well for a range of optimization problems.”
Large-scale matrix factorization with distributed stochastic gradient descent (2011, methodology): “partitions of the dataset, and parameters are averaged after each pass over the data (PSGD [11, 18]) or once at the end (ISGD [17, 18, 21]).”
A Functional Approximation Based Distributed Learning Algorithm (2013, background): “Convergence theory for such methods is inadequate [17, 18], which prompts us to ask: Q2.”
Understanding and Controlling User Linkability in Decentralized Learning (2018, background): “3 Decentralized Data Scenarios Distributed learning algorithms [15, 20, 42, 45, 58, 65, 76] typically assume data among the clients are (a) independent and identically distributed (IID) and (b) balanced, as can be seen in Figure 1a.”