Collaborative Deep Learning in Fixed Topology Networks

By Zhanhong Jiang, Aditya Balu, Chinmay Hegde, Soumik Sarkar. 2017

201 citations recorded by the source corpus.

There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentralized computation has not been explored sufficiently. In this context, this paper presents a new consensus-based distributed SGD (CDSGD) (and its momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed topology networks that enables data parallelization as well as decentralized computation. Such a framework can be extremely useful for learning agents with access to only local/private data in a communication constrained environment. We analyze the convergence properties of the proposed algorithm with strongly convex and nonconvex objective functions with fixed and diminishing step sizes using concepts of Lyapunov function construction. We demonstrate the efficacy of our algorithms in comparison with the baseline centralized SGD and the recently proposed federated averaging algorithm (that also enables data parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Federated & Distributed Learning

Citations in the corpus

Communication Efficient Decentralized Training with Multiple Local Updates (2019, methodology): “A lot of previous efforts study the communication-before update rule, including [10, 20].”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “Decentralized training (Jiang et al., 2017; Lian et al., 2017; 2018; Assran et al., 2019) reduces idling due to blocking and stragglers by employing approximate gradient aggregation (e.”
Robust and Communication-Efficient Collaborative Learning (2019, background): “Recently, there have been some works which study non-convex decentralized consensus optimization and establish convergence to a stationary point (Zeng and Yin, 2018; Hong et al., 2017, 2018; Sun and Hong, 2018; Scutari et al., 2017; Scutari and Sun, 2018; Jiang et al., 2017; Lian et al., 2017a).”
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, background): “subsequently applied it to distributed deep learning in both synchronous [9], [10], [11] and asynchronous settings [12], [13].”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “For example, (Lian et al., 2017; Jiang et al., 2017; Assran et al., 2018) consider decentralized parallel SGD where global gradient aggregations 1If an algorithm has O(1/ √ T ) convergence, then it takes 1/ 2 iterations to attain an O( ) accurate solution.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “The decentralized SGD algorithm D-PSGD (also referred as consensus-based distributed SGD), was proposed by (Jiang et al., 2017; Lian et al., 2017).”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “One notable method called decentralized parallel SGD (D-PSGD) is studied in (Lian et al. 2017)(Jiang et al. 2017) (Lian et al. 2018).”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “Recently proposed distributed SGD frameworks such as Elastic-averaging [27, 2], Federated Learning [17, 23] and decentralized SGD [15, 12] save this communication cost by allowing worker nodes to perform local updates to the parameter x instead of just computing gradients.”
Efficient Decentralized Deep Learning by Dynamic Model Averaging (2018, neutral): “The approach is compared to periodically communicating schemes, including Federated Averaging [25], a state-of-the-art approach for decentralized deep learning—more recent approaches are interesting from a theoretical perspective but show no practical improvement [13], or tackle other aspects of fed”
On the Convergence of Local Descent Methods in Federated Learning (2019, methodology): “We make the following standard assumption about the mixing matrix that is also used in analysis of standard networked distributed optimization algorithms [12,22,33].”
Asynchronous Decentralized Online Learning (2021, background): “In general, there are two major types of gossiping strategies in decentralized algorithms, namely symmetric gossiping [13, 17] and asymmetric gossiping [19, 3].”
Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning (2018, background): “Byzantine-robust distributed learning Solving large scale learning problems in distributed systems has received much attention in recent years, where communication efficiency and Byzantine robustness are two important topics (Shamir et al., 2014; Lee et al., 2015; Yin et al., 2018b; Blanchard et al.”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “First, a more connected topology leads to faster convergence in terms of iterations or communication rounds, as quantified by classic worst-case convergence bounds in terms of the spectral properties of the topology [75, 24, 89, 90, 103, 40].”
Communication-Efficient Local Decentralized SGD Methods (2019, methodology): “3 [8] Zhanhong Jiang, Aditya Balu, Chinmay Hegde, and Soumik Sarkar.”
Distributed Non-Convex First-Order optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms (2018, methodology): “References [22, 26, 27] developed methods for distributed stochastic zeroth and/or first-order non-convex optimization.”