Evaluating Differentially Private Machine Learning in Practice

By Bargav Jayaraman, David E. Evans. 2019

613 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “, 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, background): “measured privacy loss and the upper-bound ε DP guarantees— with that gap growing (exponentially) as ε becomes very large [27].”
Auditing Differentially Private Machine Learning: How Private is Private SGD? (2020, methodology): “In this section we describe our main statistical procedure for obtaining lower bounds on the privacy parameter for a given algorithm A , which functions diﬀerently from the membership inference attack used in prior work ([SSSS17, JE19] and described in Appendix D).”
Privacy in Deep Learning: A Survey (2020, background): “Jayaraman et al [125] apply membership and attribute inference attacks on multiple differentially private machine learning and deep learning algorithms, and compare their performance.”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, background): “Research like Carlini et al. (2018) and Jayaraman & Evans (2019) shows that for many problems a large gap exists between empirical measures and DP’s measure of privacy risk.”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “Thus, DP implementations can be complemented by direct measurement of memorization, as in [2, 7, 13].”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, background): “These works, although provably privacy-preserving, are not robust to poisoning attacks and produce models with undesirably poor privacy-utility tradeoffs [27].”
Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation (2020, methodology): “Estimating lower bounds through membership inference attacks: We use the membership inference attack to measure the privacy leakage of a model [45], [46], [47], [48].”
Measuring Forgetting of Memorized Training Examples (2022, background): “In this way our definitions captures the common strategy of “empirically testing” [38, 36, 57] how well differential privacy guarantees hold.”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, background): “While posthoc privacy auditing (Jagielski et al., 2020; Jayaraman & Evans, 2019; Nasr et al., 2021) and leakage mitigation processes (Majmudar et al., 2022) can be used to limit the risk, users should in general be mindful when making decisions on how and when large language models such as AlexaTM…”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, neutral): “Many works attempt empirical audits to verify lower bounds on privacy parameters (Shokri et al., 2017; Steinke et al., 2023; Jayaraman & Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).”
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning (2021, background): “also consider the relationship between privacy upper bounds and lower bounds, but do not construct as strong lower bounds as we do [27].”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “Several other modifications of the teacher-student idea were proposed recently, often referred to as “mimic learning” (Boulemtafes et al., 2020) Please refer to Jayaraman & Evans (2019) for an additional in-depth overview and comparison of various methods of DP Training.”
Detecting Pretraining Data from Large Language Models (2023, background): “…in quantifying privacy vulnerabilities within machine learning models and in verifying the accurate implementation of privacy-preserving mechanisms (Jayaraman & Evans, 2019; Jagielski et al., 2020; Zanella-Béguelin et al., 2020; Nasr et al., 2021; Huang et al., 2022; Nasr et al., 2023; Steinke et…”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “A common design pattern for a lot of supervised learning attacks is the use of shadowmodels andmeta-models or attack-models [2, 21, 35, 40, 71, 77–79, 85, 92].”
AN EMPIRICAL STUDY (2022, neutral): “This lower range is comparable to values of (real) used in building differentially-private variants of SGD [25]6.”
DarkneTZ: towards model privacy at the edge using trusted execution environments (2020, background): “MIAs can have severe privacy consequences [33, 47] motivating a number of research works to focus on tackling them [1, 28, 35].”