The Natural Auditor: How To Tell If Someone Used Your Words To Train Their Model

By Congzheng Song, Vitaly Shmatikov. 2018

18 citations recorded by the source corpus.

To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we propose a new \emph{model auditing} technique that enables users to check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, including word prediction and dialog generation. These models are at the core of many popular online services. Furthermore, they are often trained on very sensitive personal data, such as users' messages, searches, chats, and comments. We design and evaluate an effective black-box auditing method that can detect, with very few queries to a model, if a particular user's texts were used to train it (among thousands of other users). In contrast to prior work on membership inference against ML models, we do not assume that the model produces numeric confidence values. We empirically demonstrate that we can successfully audit models that are well-generalized and not overfitted to the training data. We also analyze how text-generation models memorize word sequences and explain why this memorization makes them amenable to auditing.

Subfields: Artificial Intelligence, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, background): “Further work shows how to use membership inference attacks to determine if a model was trained by using any individual user’s personal information [45].”
Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “These attacks can be seen as privacy risk analysis tools [67], which help reveal how much the model has memorized the individual samples in its training set, and what the risk of individual users is [85, 69, 59, 81, 88].”
Dynamic Backdoor Attacks Against Machine Learning Models (2020, background): “del’s parameters. Beside the above, there are multiple other types of attacks against machine learning models, such as membership inference [39], [16], [13], [34], [35], [24], [14], [25], [50], [27], [41], [37], [28], model stealing [44], [31], [46], model inversion [8], [7], [15], propoerty inferen”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, background): “While prior work has investigated issues at the intersection of neural networks and privacy (Song and Shmatikov, 2018; Salem et al., 2019; Fredrikson et al., 2015), we are unaware of work that speciﬁcally focuses on attacking the modern Transformer encoders widely used in NLP (e.g., BERT) trained…”
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning (2019, methodology): “to our attacks). More recently, multiple membership inference attacks have been proposed with new attacking techniques or targeting on different types of ML models [12], [18], [19], [22], [23], [29], [33], [40]. In theory, membership inference attack can be used to reconstruct the dataset, similar t”