Differentially Private Mixture of Generative Neural Networks

By Gergely Ács, Luca Melis, Claude Castelluccia, Emiliano De Cristofaro. 2017

138 citations recorded by the source corpus.

Generative models are used in an increasing number of applications that rely on large amounts of contextually rich information about individuals. Owing to possible privacy violations, however, publishing or sharing generative models is not always viable. In this paper, we introduce a novel solution for privately releasing generative models and entire high-dimensional datasets produced by these models. We model the generator distribution of the training data by a mixture of k generative neural networks. These are trained together and collectively learn the generator distribution of a dataset. Data is divided into k clusters, using a novel differentially private kernel k-means, then each cluster is given to separate generative neural networks, such as Restricted Boltzmann Machines or Variational Autoencoders, which are trained only on their own cluster using differentially private gradient descent. We evaluate our approach using the MNIST dataset and a large Call Detail Records dataset, showing that it produces realistic synthetic samples, which can also be used to accurately compute arbitrary number of counting queries.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Privacy in Deep Learning: A Survey (2020, methodology): “DPSGD [22] # # MNIST, CIFAR-10 Image Classification w/ DNN DP LSTM [77] # # Reddit Posts Language Model w/ LSTMs DP LOR [78] # # Artificial Data Logistic Regression DP ERM [79] # # Adult, KDD-99 Classification w/ ERM DP GAN [80] # # MNIST, MIMIC-III Data Generation w/ GAN DP GM [81] # # MNIST, CDR, ”
Differentially Private Data Generative Models (2018, background): “In addition, there has been recent work on making generative neural networks differentially private [10].”
Generating synthetic data in finance: opportunities, challenges and pitfalls (2020, background): “[41] and [42] extend differentially private synthetic data generation to the domain of unstructured data such as images and audio.”
Locally Differentially Private Data Collection and Analysis (2019, background): “One of the many topics in DP research is differentially private empirical risk minimization for machine learning [18, 10], especially for deep neural networks [1, 40, 59, 2, 53].”
Machine Learning for Synthetic Data Generation: a Review (2023, methodology): “…[40] NLP RL LSTM SynBench [41] NLP conditional Gaussian mixture CIFAR10 RelGAN [42] image and text GAN CNN COCO Image Captions dataset DPGM [43] audio and text generative artificial neural networks differentially private kernel k-means MNIST, anonymized Call Detail Record (CDR) WaveGAN…”
Synthetic Data for Deep Learning (2019, methodology): “[6] proposed to use a differentially private clustering method to split the data into k clusters, then train a separate generative models (the authors tried VAE) on their own clusters, and then create a mixture of the resulting models that will inherit differential privacy properties as well.”
Differentially Private Releasing via Deep Generative Model (2018, methodology): “Our work however differs from [14] in significant ways.”
GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models (2019, background): “The generative model, on the other hand, has also gained increasing attention and has delivered appealing applications including photorealistic image synthesis [20, 44, 52, 70], text and sound generation [4, 49, 64, 65], sanitized dataset generation [2, 7, 35, 66, 73], etc.”
Privacy Preserving Synthetic Data Release Using Deep Learning (2018, background): “For BreastCancer and Diabetes, DP-VAE has better performance; however, it fails to classify any instance in the GermanCredit dataset.”