Federated Learning: Strategies for Improving Communication Efficiency
By Jakub Konecný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, D. Bacon. 2016
5,508 citations recorded by the source corpus.
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.
Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Efficiency & Compression
- Song Han
- Peter Richtárik
- Martin Jaggi
- Sebastian U. Stich
- Dan Alistarh
- Medhi Bennis
- Dacheng Tao
- Ji Liu
In Optimization & Training
- Peter Richtárik
- Sebastian U. Stich
- Martin Jaggi
- Jakub Konecný
- Nicolas Loizou
- Jianyu Wang
- Aryan Mokhtari
- Gauri Joshi
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “However, it will always be the case that there are applications and scenarios where using a smaller or less expensive model is helpful, for example when performing client-side inference or federated learning [Konečnỳ et al., 2015, 2016].”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Another avenue would be to explore using privacy-preserving libraries such as CrypTen3 and decentralized approaches such as federated learning (Konečný et al., 2016) to handle learning from non-public data.”
- Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Our method is complementary to the techniques for reducing communication proposed by Koneˇcn`y et al. [23].”
- Advances and Open Problems in Federated Learning (2019, methodology): “Methods combining Federated Averaging with sparsification and/or quantization of model updates to a small number of bits have demonstrated significant reductions in communication cost with minimal impact on training accuracy [245].”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “This mean aggregation rule is widely used under non-adversarial settings [16, 32, 39].”
- DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “Part-based Representation Learning involves splitting the image into several stripes to learn local representations and has achieved promising performance on computer vision tasks such as person re-identification which involves image retrieval under occlusions and partial observability.”
- NoPeek: Information leakage reduction to share activations in distributed deep learning (2020, neutral): “A modified scheme called separated-DP [48] aims to provide guarantees against reconstruction in the context of federated learning [49], a popular distributed learning method for the purpose of protecting model weights trained at individual client entities while securely aggregating the average of…”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Federated learning has become an important paradigm in large-scale machine learning where the training data remains distributed over a large number of clients, which may be mobile phones or network sensors [20, 19, 23, 24, 13].”
- SplitNN-driven Vertical Partitioning (2020, methodology): “, 2019) or Random Rotation Matrix (Konečný et al., 2017) as well as privacy preserving techniques such as Secure Aggregation Protocol (Bonawitz et al.”
- Byzantine-Resilient Secure Federated Learning (2020, methodology): “More recent approaches have focused on reducing the communication overhead, by training in a smaller parameter space [20], autotuning the parameters [21], or by utilizing coding techniques [6].”
- Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overhead (2020, background): “In machine learning, and specifically in the nascent field of federated learning (Konečnỳ et al., 2016) (see, e.”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Federated learning [65] refers to a general set of techniques for model training, performed over private data owned by individual users without compromising data privacy.”