Efficient Decentralized Deep Learning by Dynamic Model Averaging

By Michael Kamp, Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, Stefan Wrobel. 2018

131 citations recorded by the source corpus.

We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared to periodically communicating state-of-the-art approaches. Moreover, we derive a communication bound that scales well with the hardness of the serialized learning problem. The reduction in communication comes at almost no cost, as the predictive performance remains virtually unchanged. Indeed, the proposed protocol retains loss bounds of periodically averaging schemes. An extensive empirical evaluation validates major improvement of the trade-off between model performance and communication which could be beneficial for numerous decentralized learning applications, such as autonomous driving, or voice recognition and image classification on mobile phones.

Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Advantages of model averaging have been studied from an empirical point of view in [27, 7, 24, 35, 17, 20].”
Towards Federated Learning at Scale: System Design (2019, background): “Alternative proposals for FL algorithms include Smith et al. (2017); Kamp et al. (2018), which would be on the high-level compatible with the system design described here.”
LEAF: A Benchmark for Federated Settings (2018, background): “…used and yet do not provide a realistic model of a federated scenario, e.g., artiﬁcial partitions of MNIST, MNIST-fashion or CIFAR-10 [22, 13, 8, 3, 11, 32, 34]; (2) realistic but proprietary federated datasets, e.g., data from an unnamed social network in [22], crowdsourced voice commands in…”
Federated Learning for Keyword Spotting (2018, methodology): “A dynamic model averaging strategy robust to concept drift based on a local model divergence criterion was recently introduced in [11].”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, background): “2017) (Su, Chen, and Xu 2018) (Kamp et al. 2018) (Lin, Stich, and Jaggi 2018) observe that model av-”
On the Convergence of Local Descent Methods in Federated Learning (2019, background): “[13] Michael Kamp, Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, and Stefan Wrobel.”
Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, background): “The authors of [97] propose to assess the need of communication based on the performance of the averag-”
Broadband Analog Aggregation for Low-Latency Federated Edge Learning (2018, methodology): “For model averaging implementation, the update significance is measured by the model variance which indicates the divergence of a particular local model from the average across all local models [10].”
A Framework for Evaluating Gradient Leakage Attacks in Federated Learning (2020, background): “By intercepting the local gradient update shared by a client to the FL server before performing federated averaging [28,25,17], the adversary can reconstruct the local training data with high reconstruction accuracy, and hence intrudes the client privacy and deceives the FL system by sneaking into c”
Federated Learning for Healthcare Informatics (2019, background): “[52] proposed to average models dynamically depending on the utility of the communication, which leads to a reduction of communication by an order of magnitude compared to periodically communicating state-of-the-art approaches.”
Parallel Restarted SGD for Non-Convex Optimization with Faster Convergence and Less Communication (2018, background): “Various experimental results [12, 1, 11, 14, 5] observe that model averaging can provide superior performance for deep learning applications.”