A Hybrid Deep Learning Architecture for Privacy-Preserving Mobile Analytics

By Seyed Ali Osia, Ali Shahin Shamsabadi, Sina Sajadmanesh, Ali Taheri, Kleomenis Katevas, Hamid Reza Rabiee, Nicholas Donald Lane, Hamed Haddadi. 2017

266 citations recorded by the source corpus.

Internet-of-Things (IoT) devices and applications are being deployed in our homes and workplaces. These devices often rely on continuous data collection to feed machine learning models. However, this approach introduces several privacy and efficiency challenges, as the service operator can perform unwanted inferences on the available data. Recently, advances in edge processing have paved the way for more efficient, and private, data processing at the source for simple tasks and lighter models, though they remain a challenge for larger and more complicated models. In this article, we present a hybrid approach for breaking down large, complex deep neural networks for cooperative, and privacy-preserving analytics. To this end, instead of performing the whole operation on the cloud, we let an IoT device to run the initial layers of the neural network, and then send the output to the cloud to feed the remaining layers and produce the final result. In order to ensure that the user’s device contains no extra information except what is necessary for the main task and preventing any secondary inference on the data, we introduce Siamese fine-tuning. We evaluate the privacy benefits of this approach based on the information exposed to the cloud service. We also assess the local inference cost of different layers on a modern handset. Our evaluations show that by using Siamese fine-tuning and at a small processing cost, we can greatly reduce the level of unnecessary, potentially sensitive information in the personal data, thus achieving the desired tradeoff between utility, privacy, and performance.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “we also compare with a traditional CNN model, denoted astraditional, with no activation privacy, this has been studied in Osia et al [63].”
Privacy in Deep Learning: A Survey (2020, background): “nsor data. The training process regulates the encoder to disregard user-identiﬁable patterns and tunes the decoder to shape the output independently of users in the training set. Another body of work [31, 147, 29, 144] propose such schemes for computer vision tasks and preserving privacy of images. ”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, background): “For instance, techniques that learn only the features relevant to a given task [15, 42, 43] can potentially serve as the basis for “leastprivilege” collaboratively trained models.”
Occlumency: Privacy-preserving Remote Deep-learning Inference Using SGX (2019, methodology): “Various privacypreserving methods have been proposed even before DL became a hot research topic, such as encryption-based operations [61], randomized noise addition [19], k-anonymity [53], and hybrid deep learning [60].”
Private Model Compression via Knowledge Distillation (2018, background): “It is sometimes not appealing for app service providers to share their valuable and highly tuned models (Osia et al. 2017).”
Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challenges (2017, methodology): “[151] PCA & Simple Embedding PCA combined with simple embedding from deep learning is used for dimensionality reduction which reduces the communication overhead between client and server.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Researchers proposed different techniques for appropriately performing the DL model splitting, such as differential private mechanism [365], Arden (privAte infeRence framework based on DNNs) [366], and hybrid edge-cloud framework [367].”
Inference Attacks Against Collaborative Learning (2018, background): “nt sharing, reducing dimensionality, and dropout are not effective. This should motivate future work on better defenses. For instance, techniques that learn only the features relevant to a given task [15,42,43] can potentially serve as the basis for “leastprivilege” collaboratively trained models. F”
Shredder: Learning Noise Distributions to Protect Inference Privacy (2019, methodology): “wn from a Laplace distribution. More recently, Wang et al. [49] proposes data nulliﬁcation and noise injection for private inference. 7 However, unlike Shredder, they retrain the network. Osia et al. [51, 50] design a private feature extraction architecture that uses principal component analysis (PC”
Transformative effects of IoT, Blockchain and Artificial Intelligence on cloud computing: Evolution, vision, trends and open challenges (2019, background): “A Hybrid approach of running deep neural network partially over an IoT device and the cloud can lead to preserving the privacy over mobile devices [51] .”
YerbaBuena: Securing Deep Learning Inference Data via Enclave-based Ternary Model Partitioning (2018, background): “Ossia et al.[39] proposed a distributed machine learning architecture to protect the user’s input privacy.”
Deep Private-Feature Extraction (2018, background): “Current solutions to this problem, such as cryptography [5], [6], complete data isolation and local processing [7] are not efﬁcient for big data and techniques relying on deep learning [8].”
DarkneTZ: towards model privacy at the edge using trusted execution environments (2020, methodology): “Based on this, hybrid training models have been proposed which run several first layers at the client-side for feature extraction and then upload these features to the server-side for classification [40].”
Shredder: Learning Noise to Protect Privacy with Partial DNN Inference on the Edge (2019, methodology): “Non-Intrusive Intrusive + Inference Shredder MiniONN[21] CryptoNets[19], GAZELLE[48], Arden[49], DPFE[50, 51]”
Olympus: Sensor Privacy through Utility Aware Obfuscation (2018, unknown): “Our work is closely related to the recently proposed obfuscation techniques that formulate the obfuscation problem as a minimax optimization [15, 18, 32, 37, 39, 41].”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, background): “[60] introduce a hybrid user-cloud framework on the privacy issue, which utilizes a private-feature extractor as its core component and breaking down large, complex deep models for cooperative, privacy-preserving analytics.”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, methodology): “They design a client-server framework based on the Siamese architecture [408], which accommodates a feature extractor in mobile devices and correspondingly a classifier in the cloud [343].”
PrivEdge: From Local to Distributed Private Training and Prediction (2020, background): “However, the curse of dimensionality may render these approaches impractical [15].”