Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning

By Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Nicholas Carlini. 2021

303 citations recorded by the source corpus.

Differentially private (DP) machine learning allows us to train models on private data while limiting data leakage. DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D′ that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private. Hence, the purpose of privacy analysis is to upper bound the probability that any adversary could successfully guess which dataset the model was trained on.In our paper, we instantiate this hypothetical adversary in order to establish lower bounds on the probability that this distinguishing game can be won. We use this adversary to evaluate the importance of the adversary capabilities allowed in the privacy analysis of DP training algorithms.For DP-SGD, the most common method for training neural networks with differential privacy, our lower bounds are tight and match the theoretical upper bound. This implies that in order to prove better upper bounds, it will be necessary to make use of additional assumptions. Fortunately, we find that our attacks are significantly weaker when additional (realistic) restrictions are put in place on the adversary's capabilities. Thus, in the practical setting common to many real-world deployments, there is a gap between our lower bounds and the upper bounds provided by the analysis: differential privacy is conservative and adversaries may not be able to leak as much information as suggested by the theoretical bound.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “, 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, methodology): “We emphasize that in the context of ML applications, using a DP mechanism even with a large ε is practically much better for privacy than using a non-DP mechanism (Song & Shmatikov, 2019; Jagielski et al., 2020; Thakkar et al., 2020; Nasr et al., 2021).”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “…individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & Shmatikov, 2019b; Nasr et al., 2021; Shokri et al., 2017).”
Counterfactual Memorization in Neural Language Models (2021, background): “…in differential privacy, the worst case performance difference of models trained on two “neighboring” datasets (identical except a single example being held-out or replaced) quantifies the privacy guarantee of a learning algorithm [Dwork et al., 2014, Nasr et al., 2021, Jagielski et al., 2020].”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “In practice, the provable guarantees oﬀered by DP algorithms are often estimated to be rather loose (i.e., the true leakage is less than what we can mathematically compute) [70, 79, 18].”
Measuring Forgetting of Memorized Training Examples (2022, methodology): “In this way our definitions captures the common strategy of “empirically testing” [38, 36, 57] how well differential privacy guarantees hold.”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “Recent work on empirical privacy auditing has shown that a strong adversary adversary that better matches the assumptions of DP can construct attacks that are almost as successful as the lower bound ε would predict (that is, you really need a small ε to protect against these attacks) (Nasr et al., 2”
On the Importance of Difficulty Calibration in Membership Inference Attacks (2021, background): “, 2019) and has been shown experimentally to be tight (Nasr et al., 2021).”
When Machine Unlearning Jeopardizes Privacy (2020, background): “DP [9, 16, 39, 49, 55] guarantees that any single data sample in a dataset has limited impact on the output.”
Membership Inference Attacks From First Principles (2021, methodology): “Existing techniques often work by altering the training pipeline, either by injecting outlier canaries [3], or using poisoning to search for worst-case memorization [22, 45].”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “• Attack False Positive Rate (FPR) [164]([148]).”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “Inspired by recent work on auditing DP algorithms with privacy attacks (Jagielski et al., 2020; Nasr et al., 2021; Tramer et al., 2022), we carry out membership inference attacks on models trained with Algorithm 1.”
Submix: Practical Private Prediction for Large-Scale Language Models (2022, methodology): “…networks—makes an implicit assumption that the adversary also observes additional information that is not accessible if h is used as a prediction API, and in practice, it causes the accounting mechanism in DP-SGD to vastly overestimate the privacy leakage parameter (cid:15) (Nasr et al., 2021).”
Detecting Pretraining Data from Large Language Models (2023, background): “…privacy vulnerabilities within machine learning models and in verifying the accurate implementation of privacy-preserving mechanisms (Jayaraman & Evans, 2019; Jagielski et al., 2020; Zanella-Béguelin et al., 2020; Nasr et al., 2021; Huang et al., 2022; Nasr et al., 2023; Steinke et al., 2023).”
Enhanced Membership Inference Attacks against Machine Learning Models (2021, methodology): “To understand the privacy risk on worst-case training data, many recent works restrict the targets to a (heuristically-selected) subset of records [3, 18] or poisoned neighboring datasets [13, 24, 33], and observe significantly higher attack success rates.”
AN EMPIRICAL STUDY (2022, unknown): “demonstrate that the theoretical upper bounds are often loose in realistic adversarial setting such as a black-box API based adversary [35].”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, unknown): “Many works attempt empirical audits to verify lower bounds on privacy parameters (Shokri et al., 2017; Steinke et al., 2023; Jayaraman & Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, background): “While posthoc privacy auditing (Jagielski et al., 2020; Jayaraman & Evans, 2019; Nasr et al., 2021) and leakage mitigation processes (Majmudar et al., 2022) can be used to limit the risk, users should in general be mindful when making decisions on how and when large language models such as AlexaTM…”