CryptoDL: Deep Neural Networks over Encrypted Data

By Ehsan Hesamifard, Hassan Takabi, Mehdi Ghasemi. 2017

414 citations recorded by the source corpus.

Machine learning algorithms based on deep neural networks have achieved remarkable results and are being extensively used in different domains. However, the machine learning algorithms requires access to raw data which is often privacy sensitive. To address this issue, we develop new techniques to provide solutions for running deep neural networks over encrypted data. In this paper, we develop new techniques to adopt deep neural networks within the practical limitation of current homomorphic encryption schemes. More specifically, we focus on classification of the well-known convolutional neural networks (CNN). First, we design methods for approximation of the activation functions commonly used in CNNs (i.e. ReLU, Sigmoid, and Tanh) with low degree polynomials which is essential for efficient homomorphic encryption schemes. Then, we train convolutional neural networks with the approximation polynomials instead of original activation functions and analyze the performance of the models. Finally, we implement convolutional neural networks over encrypted data and measure performance of the models. Our experimental results validate the soundness of our approach with several convolutional neural networks with varying number of layers and structures. When applied to the MNIST optical character recognition tasks, our approach achieves 99.52\% accuracy which significantly outperforms the state-of-the-art solutions and is very close to the accuracy of the best non-private version, 99.77\%. Also, it can make close to 164000 predictions per hour. We also applied our approach to CIFAR-10, which is much more complex compared to MNIST, and were able to achieve 91.5\% accuracy with approximation polynomials used as activation functions. These results show that CryptoDL provides efficient, accurate and scalable privacy-preserving predictions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Privacy & Security

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Secondly, cryptographic methods for training deep networks [20, 21, 22] are computationally very expensive operations which makes deploying models on the server infeasible when working with sensitive data.”
Privacy in Deep Learning: A Survey (2020, methodology): “The second class focuses on making the DNN training phase private and protecting the data used for training [21, 22, 23, 24, 25].”
Federated Machine Learning (2019, background): “With the recent advances in deep learning, privacy-preserving neural networks inference is also receiving a lot of research interests[10, 11, 14, 28, 40, 52, 54].”
Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications (2018, methodology): “rge number of images and processes them together. This, in turn, might reduce the throughput of the network significantly. A recent solution based on leveled homomorphic encryption is called CryptoDL [48]. In CryptoDL, several activation functions are approximated using low-degree polynomials and me”
Security and Privacy Issues in Deep Learning (2018, methodology): “…various attempts on fully homomorphic encryption (FHE), which allows any computable function to be performed on the encrypted data, have been proposed [Van Dijk et al. 2010; Yagisawa 2015; Brakerski and Vaikuntanathan 2014; Bos et al. 2013; Hesamifard et al. 2017; Ducas and Micciancio 2015].”
Secure, privacy-preserving and federated machine learning in medical imaging (2020, methodology): “Nevertheless, HE has successfully been applied to convolutional neural networks 70 , and its benefits demonstrated in a ‘machine learning as a service’ scenario 71 , whereby data is sent over the network to be processed on an off-site server (cloud computing).”
Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference (2018, methodology): “Our baseline model (CNN-8) resembles the CNN network architecture presented in [38] for CIFAR-10, but is designed to support the larger input image sizes (scaled from 32 × 32 × 3 to 224 × 224 × 3 input).”
A Scalable Approach for Privacy-Preserving Collaborative Machine Learning (2020, background): “The first one is Homomorphic Encryption (HE) [15], which enables computations on encrypted data, and has been applied to privacy-preserving machine learning [16, 20, 17, 41, 24, 22, 38, 19].”
CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning (2019, methodology): “One is based on Homomorphic Encryption (HE) (Gentry & Boneh, 2009) which allows for computation to be performed over encrypted data, and has been used to enable privacy-preserving machine learning solutions (GiladBachrach et al., 2016; Hesamifard et al., 2017; Graepel et al., 2012; Yuan & Yu, 2014; ”
Falcon: Honest-Majority Maliciously Secure Framework for Private Deep Learning (2020, background): “on (GMW) [11,29], each with its own advantages and disadvantages. CryptoNets [12] was one of the earliest works to demonstrate the use of homomorphic encryption to perform private inference. CryptoDL [38] developed techniques that use approximate, low-degree polynomials to implement non-linear funct”
No Peek: A Survey of private distributed deep learning (2018, background): “The works in [17,19] apply these ideas in the context of deep learning.”
Privacy-Preserving Inference in Machine Learning Services Using Trusted Execution Environments (2019, background): “Recent works [2], [16], [18], [29], [31] have explored using this technique to provide secure inference on models in the setting of adversaries trying to learn information about the data.”
ABY3: A Mixed Protocol Framework for Machine Learning (2018, background): “One recent line of work uses fully homomorphic or somewhat homomorphic encryption to evaluate the model on encrypted data [14, 18, 31, 34].”
Privacy-Preserving Machine Learning With Fully Homomorphic Encryption for Deep Neural Network (2021, background): “standard activation functions with simple nonlinear polynomials [16]–[20], called the HE-friendly network.”
HEAX: High-Performance Architecture for Computation on Homomorphically Encrypted Data in the Cloud (2019, background): “Privacy-preserving MLaaS has been studied rigorously in recent years and several protocol-level and algorithmic improvements have been proposed [36, 48, 63, 46, 19, 64, 62, 61, 42].”