Parle: parallelizing stochastic gradient descent

By Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, Ameet Talwalkar. 2017

23 citations recorded by the source corpus.

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Decentralized Bayesian Learning over Graphs (2019, positive): “Lastly, our work can be viewed as a Bayesian variant of communication-efficient methods based on SGD [12–14] which also allow the agents to make several local computations and then periodically average the local models.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “The follow-up work (Chaudhari et al., 2017) combines entropy-SGD with elastic-SGD for the distributed training, which shows the better performance than mini-batch SGD at the same wall-clock time.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “Although the efficiency of EASGD and its asynchronous and periodic averaging variants has been empirically validated (Zhang et al., 2015; Chaudhari et al., 2017), its convergence analysis under general convex or non-convex objectives is an open problem.”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “Recently proposed distributed SGD frameworks such as Elastic-averaging [27, 2], Federated Learning [17, 23] and decentralized SGD [15, 12] save this communication cost by allowing worker nodes to perform local updates to the parameter x instead of just computing gradients.”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “This idea has also been successful in deep learning where Chaudhari et al. (2016) modified SGD to seek solutions in “wide minima” with low curvature to obtain improvements in generalization performance as well as convergence rate (Chaudhari et al., 2017a).”
Entropic gradient descent algorithms and wide flat minima (2020, unknown): “Parle [Chaudhari et al., 2017], combines eSGD and EASGD (with added focusing) to obtain a distributed algorithm that shows also excellent generalization performance, consistently with the results obtained in this work.”