Detailed comparison of communication efficiency of split learning and federated learning

By Abhishek Singh, Praneeth Vepakomma, Otkrist Gupta, Ramesh Raskar. 2019

222 citations recorded by the source corpus.

We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.

Subfields: Artificial Intelligence, Computer Networks and Communications, Federated & Distributed Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Martin Jaggi
2. Qiang Yang
3. Peter Richtárik
4. Jakub Konecný
5. Peter Kairouz
6. Medhi Bennis
7. Ananda Theertha Suresh
8. Sebastian U. Stich

In Computer Networks and Communications

1. Medhi Bennis
2. Jihong Park
3. Merouane Debbah
4. Anis Elgabli
5. Sumudu Samarakoon
6. Sebastian U. Stich
7. Chaouki Ben Issaid
8. Seong-Lyun Kim

In Federated & Distributed Learning

1. Qiang Yang
2. Martin Jaggi
3. Jianyu Wang
4. Gauri Joshi
5. Chaoyang He
6. Medhi Bennis
7. Yang Liu
8. Jakub Konecný

In ML Systems & Hardware

1. Medhi Bennis
2. Jihong Park
3. Jianyu Wang
4. Oluwasanmi Koyejo
5. Martin Jaggi
6. Sebastian U. Stich
7. Anis Elgabli
8. Jakub Konecný

Citations in the corpus

• Advances and Open Problems in Federated Learning (2019, background): “In several settings, the overall communication requirements of split learning and federated learning were compared in [360].”
• FedML: A Research Library and Benchmark for Federated Machine Learning (2020, positive): “Split learning achieves better communication-efficiency under several settings, as shown in [126].”
• A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “In Reference [198], secure sharing and additive perturbation are deployed in a FL method to protect the local gradients in the training process.”
• Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “For some applications, SL yields less communication overhead compared to the aforementioned periodic-update benchmark schemes by achieving much faster convergence [82], which may not always be feasible under different tasks and data sets.”
• A Federated Learning Framework for Healthcare IoT devices (2020, unknown): “Inspired by SplitNN (Vepakomma et al., 2018; Singh et al., 2019), which is initially designed to avoid sharing raw data or model details with collaborating medical institutions, we decompose a deep neural network into two parts: i) a very shallow sub-network in the IoT device that processes…”
• Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, methodology): “To this end, link censoring for negligible model updates is applied under centralized (Chen et al., 2018; Sun et al., 2019) and decentralized network topologies (Singh et al., 2019b; Elgabli et al., 2020c).”
• End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things (2020, background): “A recent study [15] compared both models in terms of communication efficiency only.”