Overlearning Reveals Sensitive Attributes

By Congzheng Song, Vitaly Shmatikov. 2019

182 citations recorded by the source corpus.

"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races\textemdash even races that are not represented in the training data\textemdash and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “…individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & Shmatikov, 2019b; Nasr et al., 2021; Shokri et al., 2017).”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, methodology): “We ﬁrst introduce this setup in its full generality, and then we demonstrate that it captures: (a) unintended memorization via membership inference attacks [6], and (b) unintended feature leakage [8].”
Information Leakage in Embedding Models (2020, background): “In supervised learning, deep representations can reveal sensitive attributes of the input as these attributes might be used as internal features for the learning task [68].”
High-Fidelity Extraction of Neural Network Models (2019, background): “This extraction enables a number of downstream attacks, including those like overlearning [17]—that involve an inspection of the model’s internal representations, to operate in the white-box threat model. b) Fidelity Extraction : Given some target distribution D F over X , and goal similarity…”
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022, background): “Besides such memorization problems, “overlearning” on simple training objectives can reveal sensitive attributes indirectly related to the learning task (Song and Shmatikov, 2020).”
Vertical Federated Learning: Concepts, Advances, and Challenges (2022, background): “Adversarial training [108], [127] and mutual information regularization [108] are proposed to safeguard sensitive attributes of training samples.”
High Accuracy and High Fidelity Extraction of Neural Networks (2019, background): “This goal enables a number of downstream attacks, including those involving inspection of the model’s internal representations like overlearning [17], to operate in the white-box threat model.”
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging (2023, background): “This suggests that they are inappropriately using these notions as the basis for tagging behaviors (Alvi, Zisserman, and Nellaker 2018; Roberts 2018; Song and Shmatikov 2020; Wang et al. 2019).”
A Survey of Privacy Attacks in Machine Learning (2020, background): “We know, for instance, that adversarial training (using adversarial samples to make the model more robust) affects membership inference [119] and that model censoring can still leak private attributes [119].”
Dataset Inference: Ownership Resolution in Machine Learning (2021, background): “For example, Song & Shmatikov (2019) showed that models trained for gender classiﬁcation also learn features predictive of ethnicity.”