On the Privacy Risks of Model Explanations

By Reza Shokri, Martin Strobel, Yair Zick. 2019

161 citations recorded by the source corpus.

Privacy and transparency are two key foundations of trustworthy machine learning. Model explanations offer insights into a model's decisions on input data, whereas privacy is primarily concerned with protecting information about the training data. We analyze connections between model explanations and the leakage of sensitive information about the model's training set. We investigate the privacy risks of feature-based model explanations using membership inference attacks: quantifying how much model predictions plus their explanations leak information about the presence of a datapoint in the training set of a model. We extensively evaluate membership inference attacks based on feature-based model explanations, over a variety of datasets. We show that backpropagation-based explanations can leak a significant amount of information about individual training datapoints. This is because they reveal statistical information about the decision boundaries of the model about an input, which can reveal its membership. We also empirically investigate the trade-off between privacy and explanation quality, by studying the perturbation-based model explanations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han Yu
2. Qiang Yang
3. Li-zhen Cui
4. Alysa Ziying Tan
5. Mingxia Liu
6. Yu Zhang
7. Mingsheng Long
8. Jianmin Wang

Citations in the corpus

• Towards Personalized Federated Learning (2021, background): “Striving for FL model explainability may also be associated with potential privacy risks from inadvertent data leakage, as demonstrated in [110] where certain gradient-based explanation methods are prone to privacy leakage.”
• Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “[279] have demonstrated that gradient-based explanations methods leak a lot of information and make the models vulnerable to membership inference attacks.”
• Membership Inference Attacks on Machine Learning: A Survey (2021, unknown): “Binary Data Classification Adult [40] 48,842 14 2 [26, 30, 45, 76, 77, 107, 122–124, 165, 174, 181, 195, 199, 205, 223, 233] Cancer [40] 699 10 2 [107, 123, 124] Diabetes [40] 768 8 2 [107] Hepatitis [40] 155 19 2 [107] German credit [40] 1,000 20 2 [107, 215] Hospital [189] 101,766 127 2 [180] UTKF”