Auditing Differentially Private Machine Learning: How Private is Private SGD?

By Matthew Jagielski, Jonathan Ullman, Alina Oprea. 2020

331 citations recorded by the source corpus.

We investigate whether Differentially Private SGD offers better privacy in practice than what is guaranteed by its state-of-the-art analysis. We do so via novel data poisoning attacks, which we show correspond to realistic privacy attacks. While previous work (Ma et al., arXiv 2019) proposed this connection between differential privacy and data poisoning as a defense against data poisoning, our use as a tool for understanding the privacy of a specific mechanism is new. More generally, our work takes a quantitative, empirical approach to understanding the privacy afforded by specific implementations of differentially private algorithms that we believe has the potential to complement and influence analytical work on differential privacy.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “, 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, methodology): “We emphasize that in the context of ML applications, using a DP mechanism even with a large ε is practically much better for privacy than using a non-DP mechanism (Song & Shmatikov, 2019; Jagielski et al., 2020; Thakkar et al., 2020; Nasr et al., 2021).”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, positive): “While the analysis of DP-SGD might not be tight, Jagielski et al. (2020) suggest that the true ε guarantee of DP-SGD is at most one order of magnitude smaller than the current analysis suggests.”
Privacy in Deep Learning: A Survey (2020, background): “compared to well-represented groups, i.e. the poor get poorer. There is a body of work that tries to experimentally measure and audit the privacy brought by differentially private learning algorithms [124, 125]. Jagielski et al. [124] investigate whether DP-SGD offers better privacy in practice than”
Counterfactual Memorization in Neural Language Models (2021, background): “…in differential privacy, the worst case performance difference of models trained on two “neighboring” datasets (identical except a single example being held-out or replaced) quantifies the privacy guarantee of a learning algorithm [Dwork et al., 2014, Nasr et al., 2021, Jagielski et al., 2020].”
Measuring Forgetting of Memorized Training Examples (2022, methodology): “We adapt privacy auditing strategies [38, 58] which consist of two components: (1) constructing a small set of examples D p (not contained in the standard training set D ) which the model will be forced to memorize in order to classify correctly, and then (2) running a privacy test A to determine…”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “Inspired by recent work on auditing DP algorithms with privacy attacks (Jagielski et al., 2020; Nasr et al., 2021; Tramer et al., 2022), we carry out membership inference attacks on models trained with Algorithm 1.”
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning (2021, methodology): “We extend the analysis to be able to reason about both pure DP where δ = 0 and the more commonly employed (ε, δ) variant, unlike prior work which assumes δ is always negligible [26].”
Detecting Pretraining Data from Large Language Models (2023, background): “…privacy vulnerabilities within machine learning models and in verifying the accurate implementation of privacy-preserving mechanisms (Jayaraman & Evans, 2019; Jagielski et al., 2020; Zanella-Béguelin et al., 2020; Nasr et al., 2021; Huang et al., 2022; Nasr et al., 2023; Steinke et al., 2023).”
AN EMPIRICAL STUDY (2022, unknown): “In general, the values of estimated by the MI attack are underestimates, as highlighted by [23], especially as we are attacking a model trained only with ‘vanilla’ SGD, corresponding to a theoretical of∞ (ignoring any inherent randomness).”
Membership Inference Attacks on Machine Learning: A Survey (2021, unknown): “MNIST [103] 70,000 28×28×1 10 [22, 23, 25–27, 30, 66, 72, 78, 107, 110, 112, 118, 123, 124, 143, 148, 161, 162, 164, 165, 174, 181, 198–200, 205, 208, 227, 237] Fashion-MNIST [219] 70,000 28×28×1 10 [41, 66, 78, 80, 93, 94, 143, 161, 165, 187]”
Extracting Training Data from Diffusion Models (2023, background): “In addition to implementing defenses, it is important for practitioners to empirically audit their models to determine how vulnerable they are in practice [36].”
Membership Inference Attacks From First Principles (2021, background): “d) Measuring training data privacy: Given a particular training scheme, a final direction of work aims to answer the question “how much privacy does this scheme offer?” Existing techniques often work by altering the training pipeline, either by injecting outlier canaries [3], or using poisoning to s”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, unknown): “Many works attempt empirical audits to verify lower bounds on privacy parameters (Shokri et al., 2017; Steinke et al., 2023; Jayaraman & Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “While initial methods for privacy auditing required training of thousands of models (Jagielski et al., 2020; Nasr et al., 2021; Lu et al., 2022), privacy auditing can be made efficient by performing the ε estimates with fewer models (Pillutla et al.”
On the Importance of Difficulty Calibration in Membership Inference Attacks (2021, background): “Moreover, the ability to infer membership can be viewed as a lower bound for the model’s degree of memorization (Yeom et al., 2018), which is useful in itself as an empirical quantiﬁer of privacy loss (Jagielski et al., 2020; Nasr et al., 2021).”