Membership Inference Attacks Against Machine Learning Models
By Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov. 2016
5,446 citations recorded by the source corpus.
We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on. We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Sergey Levine
- Ruslan Salakhutdinov
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Nicolas Papernot
- Florian Tramèr
- Dawn Song
- Reza Shokri
- Prateek Mittal
- Matthias Hein
- Bo Li
Citations in the corpus
- Quantifying Memorization Across Neural Language Models (2022, background): “For example, membership inference attacks (Shokri et al., 2017; Yeom et al., 2018) allow an adversary to detect the presence of a given example in a model’s training dataset, and other forms of data leakage permit an adversary to learn dataset properties (Ganju et al.”
- Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Similar findings have been noted in Watson et al. (2021) and Carlini et al.”
- Deduplicating Training Data Makes Language Models Better (2021, methodology): “By this, we mean the standard privacy definition that a model should not reveal anything particular to the specific dataset it was trained on, as opposed to another training dataset from a similar distribution (Shokri et al., 2017).”
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “Neural machine learning methods (Nasr et al., 2019; Shokri et al., 2017), and language models in particular (Carlini et al., 2019; 2020) can be susceptible to training data leakage, where sensitive information can be extracted from the models.”
- Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Similar privacy problems are observed for LLMs (e.g. Nasr et al., 2019; Shokri et al., 2017; Carlini et al., 2019, 2020).”
- Red Teaming Language Models with Language Models (2022, methodology): “Data leakage can be used to infer the data used in training (“membership inference”; Shokri et al., 2017; Song and Shmatikov, 2019; Nasr et al., 2019; Hisamoto et al., 2020; Carlini et al., 2021), helping adversaries clone private, commercial LMs, violating intellectual property (Ateniese et al.,…”
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “Given a neural network f (·) trained on training data X , and an instance x, it is possible to construct a membership inference attack [42] that answers the question “Is x a member of X ?”.”
- Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “Potential risks are adversaries performing membership test (to know whether an individual is in a dataset or not) [36], recovering of partially known inputs (use the model to complete an input vector with the most likely missing bits), and extraction of the training data using the model’s…”
- Unsolved Problems in ML Safety (2021, background): “There is a large ecosystem of work on addressing societal consequences of machine learning, including AI policy [43], privacy [3, 162], fairness [72], and ethics [56].”
- Advances and Open Problems in Federated Learning (2019, background): “For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
- How To Backdoor Federated Learning (2018, background): “training data, or the resulting model (which leaks the training data [45, 48, 62]).”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Specifically, an attacker could compromise the confidentiality of users’ private training or testing data via various attacks such as model inversion attacks [21, 22], membership inference attacks [40, 49, 53], and property inference attacks [1, 24].”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “This is consistent with the work [76] where the authors attained similar results.”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “…individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & Shmatikov, 2019b; Nasr et al., 2021; Shokri et al., 2017).”
- DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “Conventionally, research in privacy-aware machine learning has primarily focused on protecting training data from membership inference [4] and model inversion attacks [5], when i) training data is distributed over clients and ii) computation of training the model is out-sourced.”
- Privacy Amplification by Decentralization (2020, background): “Unfortunately, it is by now well documented that the results of local computations (such as the parameters of a machine learning model) can leak a lot of information about the data [32].”