Algorithms that remember: model inversion attacks and data protection law

By Michael Veale, Reuben Binns, Lilian Edwards. 2018

238 citations recorded by the source corpus.

Many individuals are concerned about the governance of machine learning systems and the prevention of algorithmic harms. The EU's recent General Data Protection Regulation (GDPR) has been seen as a core tool for achieving better governance of this area. While the GDPR does apply to the use of models in some limited situations, most of its provisions relate to the governance of personal data, while models have traditionally been seen as intellectual property. We present recent work from the information security literature around ‘model inversion’ and ‘membership inference’ attacks, which indicates that the process of turning training data into machine-learned systems is not one way, and demonstrate how this could lead some models to be legally classified as personal data. Taking this as a probing experiment, we explore the different rights and obligations this would trigger and their utility, and posit future directions for algorithmic governance and regulation. This article is part of the theme issue ‘Governing artificial intelligence: ethical, legal, and technical opportunities and challenges’.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

Citations in the corpus

Making AI Forget You: Data Deletion in Machine Learning (2019, background): “Furthermore, so-called model-inversion attacks have demonstrated the capability of adversaries to extract user information from trained ML models [85].”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “[202] mention that MIAs on ML models increase their risks of being classified as private personal information under the General Data Protection Regulation (GDPR) [213].”
Inverting Gradients - How easy is it to break privacy in federated learning? (2020, unknown): “For a more general discussion, see [30].”
Monitoring Misuse for Accountable 'Artificial Intelligence as a Service' (2020, background): “This section discusses some relevant legal and technical considerations, highlighting some interesting practical challenges and opportunities for future work.”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, background): “[123] emphasize the importance of protecting against model inversion attacks as a critical step towards compliance with data protection laws such as GDPR.”
A Survey of Privacy Attacks in Machine Learning (2020, background): “Veale et al. [127] made the argument that privacy attacks such as membership inference (Section 4.1) increase the risk of machine learning models being classified as personal data under European Union’s General Data Protection Regulation (GDPR) law, because they can render a person identifiable.”
Regulating ChatGPT and other Large Generative AI Models (2023, background): “Third, we highlight key data protection risks under the GDPR, with a particular focus on model inversion [27-29].”
Machine unlearning: linear filtration for logit-based classifiers (2020, background): “…in many applications, e.g. image classiﬁcation (Yeom et al., 2018; Sablayrolles et al., 2019) or natural language processing (Carlini et al., 2019), and potentially enabling an adversary to extract sensitive information from a trained model by so-called model inversion (Veale et al., 2018).”
Explainable artificial intelligence: a comprehensive review (2021, methodology): “Wang and Yeung (2016) proposed a novel Bayesian deep learning (BDL) framework to obtain combined intelligence that supported the inference of the output. The BDL estimates the uncertainty of the black-box deep learning models by either putting distributions over the model weights or by seeking strai”
A Survey of Machine Unlearning (2022, unknown): “However, it turns out that this problem cannot be completely solved due to a lack of common frameworks and resources [131, 137, 157, 159].”
Governing artificial intelligence: ethical, legal and technical opportunities and challenges (2018, unknown): “…example around liability regarding automated vehicles, the limits of current legal frameworks in dealing with ‘big data’s disparate impact’ [2] or preventing algorithmic harms [3], social justice issues related to automating law enforcement or social welfare [4], or online media consumption [5].”
Adaptive Machine Unlearning (2021, background): “These are not necessarily safe; it is known that individual training data can be exﬁltrated from models trained in standard ways via model inversion attacks [Shokri et al., 2017, Veale et al., 2018, Fredrikson et al., 2015].”