Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

By Matt Fredrikson, Somesh Jha, Thomas Ristenpart. 2015

3,376 citations recorded by the source corpus.

Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Nicholas Carlini
  5. Quoc V. Le
  6. Bernt Schiele
  7. Nicolas Papernot
  8. Dacheng Tao

In Radiology, Nuclear Medicine and Imaging

  1. Reza Shokri
  2. Congzheng Song
  3. Nicolas Papernot
  4. Florian Tramèr
  5. Boi Faltings
  6. Aleksei Triastcyn
  7. Milad Nasr
  8. Bingsheng He

In Safety, Robustness & Interpretability

  1. Nicholas Carlini
  2. Nicolas Papernot
  3. Florian Tramèr
  4. Reza Shokri
  5. Dawn Song
  6. Prateek Mittal
  7. Bo Li
  8. Yang Zhang

Citations in the corpus