Distributed learning of deep neural network over multiple agents
By Otkrist Gupta, Ramesh Raskar. 2018
776 citations recorded by the source corpus.
In domains such as health care and finance, shortage of labeled data and computational resources is a critical issue while developing machine learning algorithms. To address the issue of labeled data scarcity in training and deployment of neural network-based systems, we propose a new technique to train deep neural networks over several data sources. Our method allows for deep neural networks to be trained using data from multiple entities in a distributed fashion. We evaluate our algorithm on existing datasets and show that it obtains performance which is similar to a regular neural network trained on a single machine. We further extend it to incorporate semi-supervised learning when training with few labeled samples, and analyze any security concerns that may arise. Our algorithm paves the way for distributed training of deep neural networks in data sensitive applications when raw data may not be shared directly.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Yang Liu
- Murali Annavaram
- Chaoyang He
- Amir Salman Avestimehr
- Praneeth Vepakomma
- Ramesh Raskar
- Medhi Bennis
In Radiology, Nuclear Medicine and Imaging
- Praneeth Vepakomma
- Chaoyang He
- Murali Annavaram
- Ramesh Raskar
- Tristan Swedish
- Abhimanyu Dubey
- Abhishek Singh
- Chandra Thapa
In Federated & Distributed Learning
- Qiang Yang
- Murali Annavaram
- Chaoyang He
- Yang Liu
- Praneeth Vepakomma
- Ramesh Raskar
- Amir Salman Avestimehr
- Jihong Park
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, neutral): “In contrast with the previous settings which focus on data partitioning and communication patterns, the key idea behind split learning [190, 393]3 is to split the execution of a model on a per-layer basis between the clients and the server.”
- DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “Part-based Representation Learning involves splitting the image into several stripes to learn local representations and has achieved promising performance on computer vision tasks such as person re-identification which involves image retrieval under occlusions and partial observability.”
- NoPeek: Information leakage reduction to share activations in distributed deep learning (2020, background): “1) Split Learning: This attack model is highly relevant to a popular resource and communication efficient variant of distributed deep learning called split learning [17, 18, 19].”
- SplitNN-driven Vertical Partitioning (2020, methodology): “We utilize SplitNN (Gupta & Raskar, 2018) architecture as a baseline architecture for distributed private training.”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “Split learning is computing and memory-efficient variant of FL introduced in [41, 42] where the model is split at a layer and the parts of the model preceding and suc-ceeding this layer are shared across the worker and server, respectively.”
- Privacy in Deep Learning: A Survey (2020, background): “Split-learning is an execution model where the neural network is split, between the client and the server [124].”
- Detailed comparison of communication efficiency of split learning and federated learning (2019, background): “Federated learning (McMahan et al., 2016; Koneˇcn`y et al., 2016) and Split learning (Gupta & Raskar, 2018; Vepakomma et al., 2018, 2019) are two methods which allow to train a model collectively from various distributed data sources without sharing raw data.”
- Split learning for health: Distributed deep learning without sharing raw patient data (2018, methodology): “In tables 1 and 2 we share more comparisons from [32] on computing resources in TFlops and communication bandwidth in GB required by these techniques.”
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, methodology): “Model parallelism-based split learning [12, 13], as shown in Fig.”
- An overview of deep learning in medical imaging focusing on MRI (2018, background): “s less relevant. However, the general deep learning community are focusing a lot of attention on the issue of privacy, and new techniques and frameworks for federated learning [333]50, split learning [334, 335] and dierential privacy [336, 337, 338] are rapidly improving. See [339] for a recent surv”
- A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, background): “Inter-layer values, i.e., activations and gradients exchange occurs between a certain participant and the central entity [31, 43, 96].”
- End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things (2020, background): “SplitNN is a recently introduced distributed learning technique [11], [12].”
- Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “Model parallelism-based split learning [11, 12] attempts to break the computational constraint, but it requires frequent communication with the server.”
- SplitFed: When Federated Learning Meets Split Learning (2020, methodology): “Split learning (Vepakomma et al. 2018; Gupta and Raskar 2018) splits a deep learning network W into multiple portions, and these portions are processed and computed on different devices.”
- Communication-Efficient Multimodal Split Learning for mmWave Received Power Prediction (2020, methodology): “Exploiting a split NN architecture [8], without sharing raw data, MultSL combines RF and image modalities by only exchanging NN activations and gradients (Fig.”
- Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, neutral): “retains a part of the neural network, and the network parts of all devices constitute a complete model [77].”
- Federated Learning for Healthcare Informatics (2019, positive): “[103] built several configurations upon a distributed deep learning method called SplitNN [36] to facilitate the health entities collaboratively training deep learning models without sharing sensitive raw data or model details.”
- A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “Split learning is a distributed learning concept, which enables collaborative machine learning without the exchange of datasets with a central server [67, 177].”
- Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “…network architecture with data partition among edge devices) [33], [34], swarm learning (i.e., decentralized device-to-device (D2D) communication architecture without central authority) [35], and split learning (i.e., model parameters partitioned among edge devices and edge servers) [36], [37].”
- R EDUCING LEAKAGE IN DISTRIBUTED DEEP LEARNING FOR SENSITIVE HEALTH DATA (2019, methodology): “Distributed deep learning methods: Split learning (Gupta & Raskar, 2018; Vepakomma et al., 2018a) is a recently developed resource efficient method for distributed deep learning by sending intermediate representations (smashed data) of split layer to another entity which completes rest of the…”