Model Reconstruction from Model Explanations

By Smitha Milli, Ludwig Schmidt, Anca Dragan, Moritz Hardt. 2018

208 citations recorded by the source corpus.

We show through theory and experiment that gradient-based explanations of a model quickly reveal the model itself. Our results speak to a tension between the desire to keep a proprietary model secret and the ability to offer model explanations. On the theoretical side, we give an algorithm that provably learns a two-layer ReLU network in a setting where the algorithm may query the gradient of the model with respect to chosen inputs. The number of queries is independent of the dimension and nearly optimal in its dependence on the model size. Of interest not only from a learning-theoretic perspective, this result highlights the power of gradients rather than labels as a learning primitive. Complementing our theory, we give effective heuristics for reconstructing models from gradient explanations that are orders of magnitude more query-efficient than reconstruction attacks relying on prediction interfaces.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Model extraction may be accelerated in presence of additional information, such as gradients of outputs w.r.t. inputs 4 [126], or contrastive explanations [7].”
Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks (2019, background): “, 2018; Wang & Gong, 2018), recover exact parameters (Lowd & Meek, 2005; Tramèr et al., 2016; Milli et al., 2018), or extract the functionality (Correia-Silva et al.”
High-Fidelity Extraction of Neural Network Models (2019, methodology): “…labels Milli et al. [19] (theoretical) Direct Recovery Neural network (2 layer) Functionally Equivalent Gradients, logits Milli et al. [19] Learning Linear model, neural network Task Accuracy Gradients Pal et al. [15] Active learning Neural network Fidelity Probabilities, labels…”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “…has explored using data augmentation techniques to reduce this hunger for data (Tang et al., 2019; Liang et al., 2020; Srinivas and Fleuret, 2018; Milli et al., 2019), we propose an alternative approach: we reduce the need for large unlabeled data by distilling not just labels but also the…”
On the Privacy Risks of Model Explanations (2019, unknown): “[27] show that gradient-based explanations can be used to reconstruct the underlying model; this serves as additional evidence of the vulnerability of transparency reports.”
The Bouncer Problem: Challenges to Remote Explainability (2019, methodology): “Milli et al. [19] provide a theoretical ground for reconstructing a remote model (a two-layer ReLu neural network) from its explanations and input gradients; if further research prove the approach practical for current applications, this technique may help to infer the use of discriminatory…”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “Some, mostly theoretical, work has demonstrated the ability to perform direct model extraction beyond linear models [50, 86].”
High Accuracy and High Fidelity Extraction of Neural Networks (2019, methodology): “As in [19], we only aim to extract neural networks with one hidden layer using the ReLU activation function.”
Survey of Attacks and Defenses on Edge-Deployed Neural Networks (2019, background): “These activations also need to be encrypted, because even in cases when the device manufacturer is not concerned about privacy, these activations can be used in order to infer the model weights [56].”
Thieves on Sesame Street! Model Extraction of BERT-based APIs (2019, background): “Model extraction attacks have been studied both empirically (Tramèr et al., 2016; Orekondy et al., 2019a; Juuti et al., 2019) and theoretically (Chandrasekaran et al., 2018; Milli et al., 2019), mostly against image classification APIs.”
Entangled Watermarks as a Defense against Model Extraction (2020, methodology): “This has been demonstrated against linear models [4, 24, 26, 38], decision trees [38], and neural networks [7, 30–32].”
Federating Recommendations Using Differentially Private Prototypes (2020, background): “…such systems do not share data directly, the model updates sent to the server may contain suﬃcient information to uncover model features and raw data information (Milli et al., 2019; Koh and Liang, 2017; Carlini et al., 2019; Hitaj et al., 2017), possibly leaking information about the users.”
Privacy Risks of Explaining Machine Learning Models (2019, background): “Indeed, in a recent paper, Milli et al. [19] show that gradient-based model explanations can be used to reconstruct the underlying model with high accuracy; their work serves as additional evidence that transparency reports are vulnerable to inference attacks.”