No Peek: A Survey of private distributed deep learning

By Praneeth Vepakomma, Tristan Swedish, Ramesh Raskar, Otkrist Gupta, Abhimanyu Dubey. 2018

113 citations recorded by the source corpus.

We survey distributed deep learning models for training or inference without accessing raw data from clients. These methods aim to protect confidential patterns in data while still allowing servers to train models. The distributed deep learning methods of federated learning, split learning and large batch stochastic gradient descent are compared in addition to private and secure approaches of differential privacy, homomorphic encryption, oblivious transfer and garbled circuits in the context of neural networks. We study their benefits, limitations and trade-offs with regards to computational resources, data leakage and communication efficiency and also share our anticipated future trends.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SplitNN-driven Vertical Partitioning (2020, methodology): “Furthermore, using SplitNN (Gupta & Raskar, 2018; Vepakomma et al. , 2018b; Sharma et al. , 2019a) also enables the use of distributed sources of data, which results in improved and generalised robust models.”
ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representations (2019, methodology): “…and we experimentally show that we can achieve similar performance as Sharma et al. (2019) to match the task (coarseLevel) assignment and reasonably good performance for class (fine-grained) assignment in a Split Learning distributed computation framework (Vepakomma et al. , 2018b, 2019, 2018a).”
ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries (2019, methodology): “Thanks to modern cloud infrastructure and distributed learning methods (Vepakomma et al. , 2018b, 2019, 2018a; Chang et al. , 2018), “expert model hubs” can make inference available to any remote client connected to the internet.”
Distilling On-Device Intelligence at the Network Edge (2019, background): “Privacy preserving aspects of FML frameworks are studied in [3], [4].”
Federated Learning on Non-IID Data: A Survey (2021, background): “Analyses of threats and additional privacy preservation techniques in FL are presented in [128, 81, 8, 92].”
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, methodology): “Note that CCVR is just a post-hoc method, so it can be easily combined with some privacy protection techniques [38] to further secure privacy.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “For privacy and security issues, the challenges and opportunities of FL have been reviewed in [30]–[33].”
A Survey on Distributed Machine Learning (2019, unknown): “[154].”
End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things (2020, background): “Unlike FL, in which each client trains the whole neural network, The SplitNN [11], [12] divides a neural network model into at least two sub-networks, and then trains the subnetworks, separately, on distributed parties (e.”