Differential Privacy Has Disparate Impact on Model Accuracy

By Eugene Bagdasarian, Vitaly Shmatikov. 2019

599 citations recorded by the source corpus.

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…et al. (2018); Curry and Rieser (2018); Dixon et al. (2018); Park et al. (2018); Thelwall (2018); Webster et al. (2018); Badjatiya et al. (2019); Bagdasaryan et al. (2019); Bamman et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019); Font and Costa-jussà (2019); Garg et al.…”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “These can include choices about many stages of the optimization process [237, 129]; privacy constraints [24], widely used compression techniques [109, 15, 169] and the choice hardware [273] have all been found to amplify harm on underrepresented protected attributes [28].”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, background): “Yet, training deep neural networks with strong DP guarantees comes at a significant cost in utility (Abadi et al., 2016; Yu et al., 2020; Bagdasaryan et al., 2019; Feldman, 2020).”
Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, background): “Such accuracy loss may sometimes be inevitable: for example, the task may involve heavy-tailed distributions and noise added by DP-SGD hinders visibility of examples in the tails (Feldman 2019; Bagdasaryan and Shmatikov 2019).”
Privacy in Deep Learning: A Survey (2020, background): “s of utility due to the addition of noise and clipping. Bagdasaryan et al. have demonstrated that this loss in utility is disparate across different sub-groups of the population, with different sizes [123]. They experimentally show that sub-groups with less training samples (less representation) los”
Salvaging Federated Learning by Local Adaptation (2020, background): “Differential privacy disproportionately reduces model accuracy for underrepresented participants (Bagdasaryan et al., 2019).”
Fair decision making using privacy-protected data (2019, background): “In addition, very recently, Bagdasaryan and Shmatikov [7] have shown the disparate impact of differential privacy on learned models.”
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation (2020, background): “This scenario is not hypothetical as it is already known that differential privacy (which formally limits the ability to memorize data) has such disparate effect on model accuracy [4].”
Does learning require memorization? a short tale about a long tail (2019, positive): “This view also explains the recent empirical results showing that the decrease in accuracy is larger for less well represented subpopulations [5].”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “Filtering for toxicity may have unexpected effects on representation in datasets or the bias of downstream outputs [40, 48, 6].”
A Review on Fairness in Machine Learning (2022, background): “Most of these studies have investigated the case of a centralized setting, in which all the dataset is held by a single party [10, 52, 100, 106, 153, 210].”
Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, background): “However, it is also known that DP greatly deceives the performance of the model under circumstances of data imbalance [105, 106], which is expected to happen in most federated scenarios.”
Datamodels: Predicting Predictions from Training Data (2022, background): “Moreover, the impact on generalization from DP is disparate across subgroups (Bagdasaryan et al., 2019).”
Privacy Regularization: Joint Privacy-Utility Optimization in LanguageModels (2021, unknown): “Differential privacy has disparate impact on the accuracy of different subgroups of the dataset (Bagdasaryan et al., 2019).”
Survey of Personalization Techniques for Federated Learning (2020, background): “Bagdasaryan et al. [13] state that the cost of differential privacy mechanisms is the reduction in accuracy, and this cost is borne unequally by clients with the underrepresented or tail participants being affected the worst.”
PrivEdge: From Local to Distributed Private Training and Prediction (2020, background): “In summary, state-of-the-art private training methods using FHE, MPC or differential privacy have high computational and communications costs, and reduce the model accuracy [54].”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, background): “…group or for each split of a challenge dataset (Hovy and Søgaard, 2015; Rudinger et al., 2018; Zhao et al., 2018; Garimella et al., 2019; Sap et al., 2019; Bagdasaryan et al., 2019; Stafanoviˇcs et al., 2020; Tan et al., 2020; Mehrabi et al., 2020; Nadeem et al., 2020; Cao and Daumé III, 2020).”