Gossip training for deep learning

By Michael Blot, David Picard, Matthieu Cord, Nicolas Thomé. 2016

124 citations recorded by the source corpus.

We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information between different threads inspired by gossip algorithms and showing good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. We compared our method to the recent EASGD in \cite{elastic} on CIFAR-10 show encouraging results.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanlin Tang
2. Ji Liu
3. Ce Zhang
4. Dan Alistarh
5. Yuanming Shi
6. Cédric Renggli
7. Khaled Ben Letaief
8. Xu Chen

In Computer Networks and Communications

1. Khaled Ben Letaief
2. Xu Chen
3. Hanlin Tang
4. Zhi Zhou
5. Liekang Zeng
6. Ji Liu
7. Yuanming Shi
8. En Li

In Electrical and Electronic Engineering

1. Khaled Ben Letaief
2. Yuanming Shi
3. Hanlin Tang
4. Ji Liu
5. Ce Zhang
6. Liekang Zeng
7. Jiasi Chen
8. Chinmay Hegde

Citations in the corpus

• Distributed Learning over Unreliable Networks (2018, background): “Reference Blot et al. (2016) considers a more radical approach, called GoSGD, where each worker exchanges gradients with a random subset of other workers in each round.”
• Collaborative Deep Learning in Fixed Topology Networks (2017, background): “In contrast, deep learning with decentralized computation can be achieved via gossip SGD algorithms [6, 7], where agents communicate probabilistically without the aid of a parameter server.”
• Deep Learning With Edge Computing: A Review (2019, background): “[131] proposed an asynchronous algorithm for gossip-based training of deep learning.”
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent (2018, background): “[33] which use random gossip suffer from communication imbalance, poor diffusion of gradients, lack of data shuffle and asynchronous exchange of gradients.”
• Federated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks (2019, methodology): “In [17] and [18], a gossip protocol is adopted for ML systems.”
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “Inspired by this, Gossip SGD (GoSGD) [64] is proposed to train DNN models in an asynchronous and decentralized way.”
• Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “…mitigation via coded computing [93], [94], [95], [96], [97], [98], [99], [100], [101] • Training in decentralized system mode [102], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112] Model Partition Based Edge Training Systems • Model partition across a large number of nodes to…”
• Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence (2019, unknown): “[71] proposed GoSGD to train DNNs asynchronously.”