Knockoff Nets: Stealing Functionality of Black-Box Models

By Tribhuvanesh Orekondy, Bernt Schiele, Mario Fritz. 2018

665 citations recorded by the source corpus.

Machine Learning (ML) models are increasingly deployed in the wild to perform a wide range of tasks. In this work, we ask to what extent can an adversary steal functionality of such ``victim'' models based solely on blackbox interactions: image in, predictions out. In contrast to prior work, we study complex victim blackbox models, and an adversary lacking knowledge of train/test data used by the model, its internals, and semantics over model outputs. We formulate model functionality stealing as a two-step approach: (i) querying a set of input images to the blackbox model to obtain predictions; and (ii) training a ``knockoff'' with queried image-prediction pairs. We make multiple remarkable observations: (a) querying random images from a different distribution than that of the blackbox training data results in a well-performing knockoff; (b) this is possible even when the knockoff is represented using a different architecture; and (c) our reinforcement learning approach additionally improves query sample efficiency in certain settings and provides performance gains. We validate model functionality stealing on a range of datasets and tasks, as well as show that a reasonable knockoff of an image analysis API could be created for as little as $30.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, methodology): “In this work, we study one possible resolution to this question: model imitation (Wallace et al., 2020; Orekondy et al., 2019).”
BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements (2020, background): “Other works have explored the threats of models themselves like backdoor attack [10, 34, 41, 42] andmodel stealing attack [13, 24, 39, 40, 43].”
Thieves on Sesame Street! Model Extraction of BERT-based APIs (2019, background): “…models contrasts with prior work, where large-scale attacks require at minimum that the adversary can access a small amount of semantically-coherent data relevant to the task (Papernot et al., 2017; Correia-Silva et al., 2018; Orekondy et al., 2019a; Pal et al., 2019; Jagielski et al., 2019).”
DAWN: Dynamic Adversarial Watermarking of Neural Networks (2019, methodology): “224 1000 100,000 - We trained MNIST-5L and GTSRB-5L models as described in [7]. We trained an additional CIFAR10-9L model, not evaluated in [7], following a similar procedure. Similarly to prior work [8], we use ResNet34 [28] architecture pre-trained on ImageNet as a basis for highcapacity models. W”
Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks (2019, methodology): “Approaches include revealing probabilities only of confident classes (Orekondy et al., 2019), rounding probabilities (Tramèr et al.”
Increasing the Cost of Model Extraction with Calibrated Proof of Work (2022, methodology): “However, PRADA does not generalize to attacks that utilize only natural data (Orekondy et al., 2019; Pal et al., 2020).”
Survey of Attacks and Defenses on Edge-Deployed Neural Networks (2019, background): “In [33], authors reframe the goal from model theft, to arriving at a ‘knockoff’ model exhibiting the”
High-Fidelity Extraction of Neural Network Models (2019, methodology): “This includes nonadaptive techniques [11, 12, 16] as well as adaptive techniques using various forms of active learning [7, 11–13, 15, 50].”
On the Difficulty of Defending Self-Supervised Learning against Model Extraction (2022, methodology): “The main goals of such an adversary are to achieve a certain task accuracy of the stolen model (Orekondy et al., 2019) or recreate a high ﬁdelity copy that can be used to mount other attacks (Jagielski et al., 2020), such as the construction of adversarial examples (Szegedy et al., 2014; Papernot…”
NN-Lock: A Lightweight Authorization to Prevent IP Threats of Deep Learning Models (2022, background): “Recent works [Orekondy et al. 2019; Tramèr et al. 2016; Yu et al. 2020b] demonstrated successful model stealing attacks against commercial MLaaS services, resulting in immense economic losses for the original model owner or service provider in practical scenarios.”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “Several other strategies for selecting the most suitable data for querying the target model use: (i) data that are not synthetic but belong to different domains such as images from different datasets [13, 72], (ii) semi-supervised learning techniques such as rotation loss [107] or MixMatch [5] to au”
Entangled Watermarks as a Defense against Model Extraction (2020, methodology): “This has been demonstrated against linear models [5,32,35,51], decision trees [51], and DNNs [8,40,41,43].”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, methodology): “, 2016), vision (Orekondy et al., 2019), and language tasks (Krishna et al.”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “Attacks that compromise privacy of models in this setting include: model stealing [52,62,77], membership inference [66,67,71] which identiﬁes if a particular example was used during training, attribute inference [28,56] to identify properties that holds true for subsets of data, and model inversion…”
High Accuracy and High Fidelity Extraction of Neural Networks (2019, background): “Learning LM, NN Task Accuracy Gradients Pal et al. [15] Active learning NN Fidelity Probabilities, labels Chandrasekharan et al. [13] Active learning LM Functionally Equivalent Labels Copycat CNN [16] Learning CNN Task Accuracy, Fidelity Labels Papernot et al. [7] Active learning NN Fidelity Labels ”
PRADA: Protecting Against DNN Model Stealing Attacks (2018, methodology): “These attacks benefit from correlations between different [40] or same [37] pre-trained models.”
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning (2019, background): “[34] propose a more advanced attack on stealing the target model’s functionality and show that their attack is able to replicate a mature commercial machine learning API.”
Amnesiac Machine Learning (2020, background): “…model extraction attacks (Jagiel-ski et al. 2019), (Tram`er et al. 2016), (Shi, Sagduyu, and Grushin 2017) have the ability to steal functionality of models even with only black-box access, and in some cases even with only access to truncated prediction vectors (Orekondy, Schiele, and Fritz 2019).”
Dataset Inference: Ownership Resolution in Machine Learning (2021, methodology): “Depending on the level of access provided by the prediction APIs, model extraction may be performed by only using the labels (Chan-drasekaran et al., 2019; Correia-Silva et al., 2018) or the entire prediction logits of the deployed service (Orekondy et al., 2018).”
IReEn: Iterative Reverse-Engineering of Black-Box Functions via Neural Program Synthesis (2020, methodology): “More recently, similar principles have been applied to the machine learning domain to find internal information or copy black-box models like deep learning architectures (Tramèr et al., 2016; Oh et al., 2018; Orekondy et al., 2019).”