GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent

By J. Daily, Abhinav Vishnu, Charles Siegel, Thomas E. Warfel, Vinay C. Amatya. 2018

107 citations recorded by the source corpus.

In this paper, we present GossipGraD - a gossip communication protocol based Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL) algorithms on large-scale systems. The salient features of GossipGraD are: 1) reduction in overall communication complexity from {\Theta}(log(p)) for p compute nodes in well-studied SGD to O(1), 2) model diffusion such that compute nodes exchange their updates (gradients) indirectly after every log(p) steps, 3) rotation of communication partners for facilitating direct diffusion of gradients, 4) asynchronous distributed shuffle of samples during the feedforward phase in SGD to prevent over-fitting, 5) asynchronous communication of gradients for further reducing the communication cost of SGD and GossipGraD. We implement GossipGraD for GPU and CPU clusters and use NVIDIA GPUs (Pascal P100) connected with InfiniBand, and Intel Knights Landing (KNL) connected with Aries network. We evaluate GossipGraD using well-studied dataset ImageNet-1K (~250GB), and widely studied neural network topologies such as GoogLeNet and ResNet50 (current winner of ImageNet Large Scale Visualization Research Challenge (ILSVRC)). Our performance evaluation using both KNL and Pascal GPUs indicates that GossipGraD can achieve perfect efficiency for these datasets and their associated neural network topologies. Specifically, for ResNet50, GossipGraD is able to achieve ~100% compute efficiency using 128 NVIDIA Pascal P100 GPUs - while matching the top-1 classification accuracy published in literature.

Subfields: Biomedical Engineering, Computational Theory and Mathematics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

Citations in the corpus

Distributed Learning over Unreliable Networks (2018, background): “Recently, Daily et al. (2018) proposed GossipGrad, a more complex gossip-based scheme with upper bounds on the time for workers to communicate indirectly, periodic rotation of partners and shuffling of the input data, which provides strong empirical results on large-scale deployments.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “As strong consistency is not required for distributed deep learning, this method has shown marginal success for SGD [55, 126, 209], attaining both convergence and faster performance than allreduce SGD up to 32 nodes.”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “[66] show that the trivial gossip-based algorithms at scale lead to a communication imbalance, poor convergence, and heavy communication overhead.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “In edge AI systems with inherently isolated data, distributed realizations of SGD will play a key role and should be carefully investigated.”
Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence (2019, background): “introduce GossipGraD, which can reduce the communication complexity greatly to ensure the fast convergence [73].”
A generic communication scheduler for distributed DNN training acceleration (2019, methodology): “, Stale Synchronous Parallel [20] for PS and different all-reduce algorithms [10, 14]; (4) minimizing network flow completion time by using flow control [26] or Coflow scheduling [13].”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, methodology): “In a ring architecture, there is no centralised server to coordinate the parameter update; instead, each node both locally computes gradients and performs parameter aggregation by communicating with other nodes using aGossip algorithm [98, 28, 69].”