Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning

By Ahmed Salem, Apratim Bhattacharyya, Michael Backes, Mario Fritz, Yang Zhang. 2019

298 citations recorded by the source corpus.

Machine learning (ML) has progressed rapidly during the past decade and the major factor that drives such development is the unprecedented large-scale data. As data generation is a continuous process, this leads to ML model owners updating their models frequently with newly-collected data in an online learning scenario. In consequence, if an ML model is queried with the same set of data samples at two different points in time, it will provide different results. In this paper, we investigate whether the change in the output of a black-box ML model before and after being updated can leak information of the dataset used to perform the update, namely the updating set. This constitutes a new attack surface against black-box ML models and such information leakage may compromise the intellectual property and data privacy of the ML model owner. We propose four attacks following an encoder-decoder formulation, which allows inferring diverse information of the updating set. Our new attacks are facilitated by state-of-the-art deep learning techniques. In particular, we propose a hybrid generative model (CBM-GAN) that is based on generative adversarial networks (GANs) but includes a reconstructive loss that allows reconstructing accurate samples. Our experiments show that the proposed attacks achieve strong performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “Training data extraction refers to the risk of partially extracting training samples by interacting with a trained language model [80, 12, 99, 13].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Both of these privacy attacks can improve when models are updated repeatedly, if updated models are released to the adversary (Salem et al., 2020; Zanella-Béguelin et al., 2020; Jagielski et al., 2022).”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, background): “The former one assumes that the attacker compromises some participated devices and uses them to achieve malicious intentions, e.g. importing backdoor to FL [14, 15], adversarial poisoning [16, 17, 18], membership or property inference [19, 20, 21, 22], and reconstruction attack [23, 24, 25].”
The Limitations of Federated Learning in Sybil Settings (2020, background): “For example, adversaries posing as honest clients can send erroneous updates to poison the trained model [3,8,15,19,23,28,36], invert or reconstruct the data of honest clients [26,41,42,48,54,56], or gain access to models without usefully contributing [34].”
Systematic Evaluation of Privacy Risks of Machine Learning Models (2020, background): “ks of machine learning models. 1 Introduction A recent thread of research has shown that machine learning (ML) models memorize sensitive information of training data, indicating serious privacy risks [4,9,10,15,35,39,40]. In this paper, we focus on the membership inference attack, where the adversar”
Membership Inference Attacks From First Principles (2021, background): “But do these attacks work at low false-positive rates? • Are attacks with extra capabilities more effective? Prior work has shown that access to gradient queries [46] or intermediate models [58] improves attack AUC.”
Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorization (2021, methodology): “Model Inversion Attack (He et al., 2019; Salem et al., 2020) is able to recover the input data via an analysis of the model prediction.”
BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements (2020, background): “For instance, multiple works have explored the security and privacy threats of data used to train DNNmodels, such as membership inference attack [30, 35, 36], dataset reconstruction attack [33], and property inference attack [8, 12].”
MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples (2019, background): “Other privacy/confidentiality attacks against ML: There exist multiple other types of privacy/confidentiality attacks against ML models [2, 14–16, 36, 44, 55, 64, 65].”
Analyzing Information Leakage of Updates to Natural Language Models (2019, background): “Relationship to other attacks on training data. are reconstruction attacks [24] against the updated model, as the The goal of membership inference attacks [25, 26] is weaker in Finally, model inversion attacks [12, 13] repurpose a model to work backwards , inferring unknown attributes of…”
Addressing Class Imbalance in Federated Learning (2020, methodology): “, methods in (Salem et al. 2020) cannot launch reconstruction attacks anymore, and it is hard for attackers to send meaningful targeted information as well.”
Do Language Models Plagiarize? (2022, background): “There is a growing body of literature that aims to study the memorization of neural LMs by recovering texts in the training corpus [31, 47] or extracting artificially injected canaries [37, 58].”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “…between the works of this category is between those that create an actual reconstruction of the data [41, 134, 138, 145, 148] and the ones that create class representatives or probable values of sensitive features that do not necessarily belong to the training dataset [28, 43, 47, 109, 138].”
Privacy Risks of General-Purpose Language Models (2020, background): “Later, some very recent works have proposed ﬁner-grained attacks which attempt to recover the exact training images or texts from the predictions [58], [77] or the gradients [47], [82] in an unknown mini-batch during the training phase.”