Functional Mechanism: Regression Analysis under Differential Privacy

By Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Y. Yang, Marianne S. Winslett. 2012

457 citations recorded by the source corpus.

e-differential privacy is the state-of-the-art model for releasing sensitive information while protecting privacy. Numerous methods have been proposed to enforce e-differential privacy in various analytical tasks, e.g., regression analysis. Existing solutions for regression analysis, however, are either limited to non-standard types of regression or unable to produce accurate regression results. Motivated by this, we propose the Functional Mechanism, a differentially private method designed for a large class of optimization-based analyses. The main idea is to enforce e-differential privacy by perturbing the objective function of the optimization problem, rather than its results. As case studies, we apply the functional mechanism to address two most widely used regression models, namely, linear regression and logistic regression. Both theoretical analysis and thorough experimental evaluations show that the functional mechanism is highly effective and efficient, and it significantly outperforms existing solutions.

Subfields: Artificial Intelligence, Statistics and Probability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Privacy & Security

Citations in the corpus

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Several works have adopted DP as an efficient defense mechanism such as [5, 7, 11, 12, 18, 23, 24, 37, 53, 60, 65, 72, 86].”
AdaCliP: Adaptive Clipping for Private SGD (2019, methodology): “In objective perturbation techniques, the objective function is perturbed by the appropriate scaling of Laplace or Gaussian noise and the machine learning model is trained over perturbed objective function [10, 11, 26].”
Wireless Network Intelligence at the Edge (2018, background): “Thus, so long as the noise level is appropriately adjusted, one can improve both accuracy and privacy, as done in [162].”
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing (2014, positive): “0 .25 1.0 5.0 20.0 100.0 1.00 1.05 1.10 1.15 1.20 1.25 1.30 Mortality, Private LR Mortality, Std. LR ε (privacy budget) Relative Risk ( Mortality) 0.60 0.65 0.70”
The Value of Collaboration in Convex Machine Learning with Differential Privacy (2019, methodology): “In [17], a large class of optimization-based DP machine learning algorithms are developed by appropriately perturbing the objective function of the ML training algorithm.”
Differential Privacy Preservation for Deep Auto-Encoders: an Application of Human Behavior Prediction (2016, methodology): “7), denoted as ̂RE(D,W ) 2) The function ̂RE(D,W ) is perturbed by using functional mechanism (FM) (Zhang et al. 2012), the perturbed function is denoted as RE(D,W ) 3) Compute W = argminW RE(D,W ) 4) Private Auto-encoder (PA) stacking 5) Derive and perturb the polynomial approximation of cross-entr”
Signal Processing and Machine Learning with Differential Privacy (2013, positive): “From these two settings, the literature has spread to cover more complex data processing algorithms such as realtime signal processing [11]–[13], classification [14]–[16], dimensionality reduction [17], [18], and auction design [19].”
Towards Making Systems Forget with Machine Unlearning (2015, neutral): “I. INTRODUCTION”
Towards Practical Differentially Private Convex Optimization (2019, neutral): “There have also been works [34], [35] on DP regression analysis, a subset of DP convex optimization.”
Differentially Private Model Publishing for Deep Learning (2019, methodology): “Differential privacy (DP), a defacto standard for privacy that offers provable privacy guarantees, has been applied for privacy-preserving machine learning [8], [10], [11], [32], [42].”
Understanding Membership Inferences on Well-Generalized Learning Models (2018, methodology): “fferential privacy [11] is a prominent way to formalize privacy against membership inference. It has been applied to various machine learning models including decision trees [19], logistic regression [41], and neural networks [4,31]. However, there are no generic methods to achieve differential priv”
Membership Inference Attacks Against Machine Learning Models (2016, background): “Differential privacy [12] has been applied to linear and logistic regression [8], [37], support vector machines [28], risk minimization [5], [9], [34], deep learning [1], [30], learning an unknown probability distribution over a discrete population from random samples [10], and releasing hyper-param”
Differentially Private Bayesian Linear Regression (2019, background): “A standard assumption in literature [Awan and Slavkovic, 2018, Sheffet, 2017, Wang, 2018, Zhang et al., 2012] is to assume x and y have known a priori lower and upper bounds, ( a x , b x ) and ( a y , b y ) , with bound widths w x = b x − a x (assuming, for simplicity, equal bounds for all…”
Interpretable and Differentially Private Predictions (2019, background): “An alternative approach is to directly perturb the objective with additive noise (Zhang et al. 2012; Phan et al. 2016; Phan, Wu, and Dou 2017).”
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting (2017, background): “Although there has been a significant amount of work aimed at developing machine learning algorithms that satisfy definitions such as differential privacy [8], [10], [11], [12], [13], [14], the factors that bring about specific types of privacy risk in applications of standard machine learning algor”
Distributed Differentially Private Averaging with Improved Utility and Robustness to Malicious Parties (2020, background): “A promising direction is to combine GOPA with the functional mechanism [69].”
Differential Privacy for Deep and Federated Learning: A Survey (2022, background): “The approaches of [66] and [67] are the same, except that [67] adds noise with the same privacy budget whichmay decrease themodel’s accuracy.”