Systematic Evaluation of Privacy Risks of Machine Learning Models

By Liwei Song, Prateek Mittal. USENIX Security Symposium, 2020

510 citations recorded by the source corpus.

Machine learning models are prone to memorizing sensitive data, making them vulnerable to membership inference attacks in which an adversary aims to guess if an input sample was used to train the model. In this paper, we show that prior work on membership inference attacks may severely underestimate the privacy risks by relying solely on training custom neural network classifiers to perform attacks and focusing only on the aggregate results over data samples, such as the attack accuracy. To overcome these limitations, we first propose to benchmark membership inference privacy risks by improving existing non-neural network based inference attacks and proposing a new inference attack method based on a modification of prediction entropy. We also propose benchmarks for defense mechanisms by accounting for adaptive adversaries with knowledge of the defense and also accounting for the trade-off between model accuracy and privacy risks. Using our benchmark attacks, we demonstrate that existing defense approaches are not as effective as previously reported. Next, we introduce a new approach for fine-grained privacy analysis by formulating and deriving a new metric called the privacy risk score. Our privacy risk score metric measures an individual sample's likelihood of being a training member, which allows an adversary to perform membership inference attacks with high confidence. We experimentally validate the effectiveness of the privacy risk score metric and demonstrate that the distribution of the privacy risk score across individual samples is heterogeneous. Finally, we perform an in-depth investigation for understanding why certain samples have high privacy risk scores, including correlations with model sensitivity, generalization error, and feature embeddings. Our work emphasizes the importance of a systematic and rigorous evaluation of privacy risks of machine learning models.

Subfields: Artificial Intelligence, Safety Research

View Paper: https://www.semanticscholar.org/paper/fd6e4d74b8df1eed41a0915c5d760fe3cfa58278

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Bag of Tricks for Training Data Extraction from Language Models (2023, methodology): “Metric-based methods, on the other hand, make membership inferences by first calculating metrics on the model prediction vectors (Yeom et al., 2018; Salem et al., 2018; Sablayrolles et al., 2019; Song & Mittal, 2021; Choquette-Choo et al., 2021).”
On the Importance of Difficulty Calibration in Membership Inference Attacks (2021, methodology): “Speciﬁcally, we considered the entropy and modiﬁed entropy scores proposed by Song & Mittal (2021) as well as the Merlin and Morgan attacks proposed by Jayaraman et al. (2021).”
Unrolling SGD: Understanding Factors Influencing Machine Unlearning (2021, methodology): “4) In § 8.4, we observe that single gradient unlearning decreases the PRS of the baseline (i.e., γ = 0).”
Membership Inference Attacks From First Principles (2021, methodology): “Given a model f and target sample (x, y), the attack computes a hypothesis test between the (per-class) member and non-member distributions (see [61]).”
Enhanced Membership Inference Attacks against Machine Learning Models (2021, background): “[28], Song and Mittal [31] studied and improved the shadow model attack methodology.”
Membership Inference Attacks on Machine Learning: A Survey (2021, methodology): “However, Song and Mittal [185] re-evaluate the effectiveness of Memguard using metric-based attacks and find that the defended models are still susceptible to membership attacks.”