Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data
By Felix Sattler, Simon Wiedemann, K. Müller, Wojciech Samek. 2019
1,679 citations recorded by the source corpus.
Federated learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods, however, are only of limited utility in the federated learning setting, as they either only compress the upstream communication from the clients to the server (leaving the downstream communication uncompressed) or only perform well under idealized conditions, such as i.i.d. distribution of the client data, which typically cannot be found in federated learning. In this article, we propose sparse ternary compression (STC), a new compression framework that is specifically designed to meet the requirements of the federated learning environment. STC extends the existing compression technique of top- $k$ gradient sparsification with a novel mechanism to enable downstream compression as well as ternarization and optimal Golomb encoding of the weight updates. Our experiments on four different learning tasks demonstrate that STC distinctively outperforms federated averaging in common federated learning scenarios. These results advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.
Subfields: Artificial Intelligence, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Jakub Konecný
- Peter Richtárik
- Virginia Smith
- Jihong Park
- Martin Jaggi
- Jianyu Wang
- H. Vincent Poor
In Federated & Distributed Learning
- Jianyu Wang
- Gauri Joshi
- Zachary B. Charles
- Dusist Niyato
- Peter Richtárik
- Jakub Konecný
- Virginia Smith
- Medhi Bennis
Citations in the corpus
- SplitNN-driven Vertical Partitioning (2020, neutral): “An interesting line of research for the future, would be to study the effect on the convergence of compression techniques such as STC (Sattler et al. , 2019) or Random Rotation Matrix (Koneˇcný et al. , 2017) as well as privacy preserving techniques such as Secure Aggregation Protocol (Bonawitz et…”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “Moreover, FedNova is also compatible with and complementary to gradient compression/quantization [43–48] and fair aggregation techniques [49, 50].”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “, 2019), and communication-efficiency (Konečný et al., 2016; Sattler et al., 2019; Basu et al., 2019; Reisizadeh et al., 2020).”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Besides, many empirical studies are investigating the effect of system heterogeneity [31, 55], the power of scalability [22], strategies of preserving privacy [6, 27], and saving communication [9].”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Nevertheless, several works have provided practical strategies in federated settings, such as forcing the updating models to be sparse and low-rank; performing quantization with structured random rotations [59]; using lossy compression and dropout to reduce server-to-device communication [15]; and a”
- Distilled One-Shot Federated Learning (2020, background): “Sparsifying the weights may perform better than FedAvg alone [28].”
- Overcoming Forgetting in Federated Learning on Non-IID Data (2019, background): “communication by sparsification and quantization of the updates sent to the central point after each local mini batch [9].”
- Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data (2020, background): “Compared with existing well-studied distributed computing, there are four main key differences (Konečnỳ et al. (2016a); Kairouz et al. (2019); Sattler et al. (2019); Li et al. (2019a,b)): 1) the number of clients N is large and the communication between clients ar X iv :2 00 9.”
- Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data (2020, methodology): “Another stream of research proposed model compression to reduce the computation cost required for the model parameter exchanges, which can be performed via several strategies, such as low-rank representation [15], model parameter quantization [15], [16], [17], update reuse [18], neural network pruni”
- Artificial Intelligence in Dentistry: Chances and Challenges (2020, background): “Various robust and communication-efficient training schemes for federated learning have been developed (Sattler et al. 2019).”
- Federated Learning on Non-IID Data: A Survey (2021, methodology): “Parametric models Non-IID data does affect the learning performance on parametric models [95, 166, 112, 129, 119, 7] and will always cause global model divergence in horizontal FL.”
- Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AI (2021, background): “…not automatically contribute to each other above manual sampling of the models and update of hyper-parameters; (ii) federated learning schemes [38,39], where a global model is constructed considering update signals of all local clients which are merged into a global model and distributed back to…”
- Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning (2019, background): “…to reduce the cost of gradient communication (Al-istarh et al., 2017; Bernstein et al., 2018; Wen et al., 2017; Yu et al., 2018; Chen et al., 2018; Wang et al., 2019; Stich et al., 2018; Suresh et al., 2017; Sattler et al., 2019; Tang et al., 2018; 2019; Assran et al., 2019; Acharya et al., 2019).”