Iterative parameter mixing for distributed large-margin training of structured predictors for natural language processing

By Gregory F. Coppola. 2015

19 citations recorded by the source corpus.

Subfields: Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Optimization & Training

Citations in the corpus

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, methodology): “A special algorithm of this type is • local SGD (Coppola, 2015; Zhou & Cong, 2018; Stich, 2019b) where averaging on the complete graph is performed every τ iterations and only local steps are performed otherwise (mixing matrix In for τ − 1 steps).”
On the Convergence of FedAvg on Non-IID Data (2019, methodology): “The prior work Coppola (2015); Zhou and Cong (2017); Stich (2018); Yu et al.”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, background): “Another approach for mitigating the communication bottlenecks is by having infrequent communication, which has been popularly referred to in the literature as iterative parameter mixing, see [21], and model averaging, see [20], [23], [30] and references therein.”
Local SGD Converges Fast and Communicates Little (2018, background): “[22] for perceptron training (iterative parameter mixing), see also [7], [5, 46, 48] for the training of deep neural networks (model averaging) or in federated learning [23].”
Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “While local SGD has been used among practitioners for a long time, see e.g. (Coppola, 2015; McDonald et al., 2010), its theoretical analysis has been limited until recently.”
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization (2019, background): “2 Another class of algorithms that are based on the idea of infrequent communication , workers do not communicate in each iteration; rather, they send the updates after performing a ﬁxed number of local gradient steps [BDKD19,Sti18,YYZ19,Cop15].”
Federated Learning under Arbitrary Communication Patterns (2021, background): “Such a strategy is commonly referred to as local SGD (Mangasarian, 1995; Zinkevich et al., 2010; Coppola, 2015; Stich, 2018; Zhou & Cong, 2018), but is also known in the literature under various other names (such as parallel SGD ) (Yu et al., 2019b).”
Better Communication Complexity for Local SGD (2019, background): “While local SGD has been popular among practitioners for a long time [5, 12], its theoretical understanding has remained elusive until very recently [1, 9, 14, 15, 16, 17] (see Table 1).”