The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets

By Nicholas Carlini, Chang Liu, Jernej Kos, Ú. Erlingsson, Dawn Song. 2018

200 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

In Privacy & Security

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
How To Backdoor Federated Learning (2018, neutral): “We note in passing that this immediately breaks privacy, which is a key objective of federated learning, because the participant’s training data can be partially reconstructed from the selected model [5, 37].) As the training is converging, models near the current global model are more likely to be ”
Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, background): “Research like Carlini et al. (2018) and Jayaraman & Evans (2019) shows that for many problems a large gap exists between empirical measures and DP’s measure of privacy risk.”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “Thus, DP implementations can be complemented by direct measurement of memorization, as in [2, 7, 13].”
Differentially Private Meta-Learning (2019, background): “Furthermore, Carlini et al. (2018) showed that deep neural networks can effectively memorize user-unique training examples, which can be recovered even after only a single epoch of training.”
Differentially Private Learning with Adaptive Clipping (2019, neutral): “However, recent work has shown that it is possible to extract detailed information about individual training examples using only the parameters of a trained model (Fredrikson et al., 2015; Wu et al., 2016; Shokri et al., 2017; Carlini et al., 2018; Melis et al., 2018).”
Federated Learning Of Out-Of-Vocabulary Words (2019, background): “While Federated Learning removes the need to upload raw user material — here OOV words — to the server, the privacy risk of unintended mem-orization still exists (as demonstrated in (Carlini et al., 2018)).”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “However, sharing other information such as model updates as part of the training process can also leak sensitive user information [8, 17, 39, 78].”
Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Input inference [20], attribute inference [21], parameter inference [22], [23], and side-channel attacks [24] are several examples of such attacks.”
Auditing Data Provenance in Text-Generation Models (2018, background): “ting its accuracy on the main task. By contrast, we demonstrate that popular text-generation models unintentionally memorize training data in their hidden activation units and outputs. Carlini et al. [7] showed that deep learning-based generative sequence models unintentionally memorize training dat”
Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “Input inference [31], attribute inference [32], parameter inference [33], [34], and side-channel attacks [35] are several examples of such attacks.”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “8] attacks force a machine learning model to memorize the training data in such a way that an adversary can later extract training inputs with only black-box access to the model. Then, Carlini et al. [11] show that deep learningbased language models trained on text data can unintentionally memorize ”
Gmail Smart Compose: Real-Time Assisted Writing (2019, methodology): “Since Smart Compose is trained on billions of phrases and sentences,similartotheway spammachinelearningmodelsaretrained, we have done extensive testing to make sure that only common phrases used by multiple users are memorized by our model, using findings from Carlini et al. [8].”
Machine Learning with Membership Privacy using Adversarial Regularization (2018, background): “These inference attacks include input inference [19], membership inference [45], attribute inference [9], parameter inference [47, 48], and side-channel attacks [50].”
Certified Data Removal from Machine Learning Models (2019, methodology): “This guarantees that membership-inference attacks (Yeom et al., 2018; Carlini et al., 2019) are unsuccessful on data that was removed from the model.”