Deep learning with Elastic Averaging SGD

By Sixin Zhang, Anna Choromanska, Yann LeCun. 2014

645 citations recorded by the source corpus.

We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Theano: A Python framework for fast computation of mathematical expressions (2016, methodology): “[23] Sixin Zhang, Anna E Choromanska, and Yann LeCun, “Deep learning with elastic averaging SGD,” in Advances in Neural Information Processing Systems (2015) pp.”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, methodology): “Indeed, similar algorithms were proposed in (Zhang et al., 2015) and (Mobahi, 2016) for Deep Learning.”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, background): “Many efforts have been made to parallelize SGD for Deep Learning (Dean et al., 2012; Das et al., 2016; Zhang et al., 2015), yet the speed-ups and scale-out are still limited by the batch size.”
Advances and Open Problems in Federated Learning (2019, background): “[49] J. Wang, A. Sahu, G. Joshi, and S. Kar, “MATCHA: Speeding up decentralized SGD via matching decomposition sampling,” Preprint, May 2019.”
Federated Optimization in Heterogeneous Networks (2018, methodology): “While the intuition is similar to EASGD (this term helps to prevent large deviations on each device/machine), EASGD employs a more complex moving average to update parameters, is limited to using SGD as a local solver, and has only been analyzed for simple quadratic problems.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, background): “EASGD (Zhang et al., 2014) periodically synchronizes the models on the workers and the servers with moving average.”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “Elastic-average SGD (Zhang et al., 2015) uses a related approach, but with a parameter server rather than ALLREDUCE.”
Privacy-Preserving Tensor Factorization for Collaborative Health Data Analysis (2019, methodology): “Using the EASGD approach, the global consensus optimization problem is solved alternatively between the local sites and the central server.”
On the Convergence of FedAvg on Non-IID Data (2019, background): “The prior work [35, 41, 28, 30, 33] assumes the data are iid generated by or partitioned among the N devices, that is, Dk = D for all k ∈ [N ].”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “…references in a chronological order are Langford et al. (2009); Niu et al. (2011); Agarwal and Duchi (2011); Paine et al. (2013); Li et al. (2014b); Zhang et al. (2014); Keuper and Pfreundt (2015); De and Goldstein (2015); Feyzmahdavian et al. (2015); Lian et al. (2015); Mania et al. (2015); Zhao…”
Wireless Network Intelligence at the Edge (2018, background): “A typical example is elastic SGD (ESGD), where the local MSI is each device’s local model weights, and the global MSI is the mean weights averaged over all devices [109].”
Distributed Learning over Unreliable Networks (2018, background): “They show that GoSGD can be faster than Elastic Averaging SGD (Zhang et al., 2015) on CIFAR-10, but provide no large-scale experiments or theoretical justification.”
Stochastic Gradient Push for Distributed Deep Learning (2018, background): “(2016) make a theoretical connection between PUSHSUMbased methods and Elastic Averaging SGD (Zhang et al., 2015).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “[29, 30], require iid distributed data, this is not required in the federated setting.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “Although the efﬁciency of EASGD and its asynchronous and periodic averaging variants has been empirically validated (Zhang et al., 2015; Chaudhari et al., 2017), its convergence analysis under general convex or non-convex objectives is an open problem.”
COLA: Decentralized Linear Learning (2018, background): “Our results are free of the restrictive assumptions made by stochastic methods [Zhang et al., 2015, Wang et al., 2017], which requires i.i.d. data distribution over all devices.”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “verage of nal individual solutions from the previous epoch. This is why we call Algorithm 1 \parallel restarted SGD". Such an algorithm is dierent from elastic averaging SGD (EASGD) proposed in [16] which periodically drags each local solution towards their average using a controlled weight. No”