Thieves on Sesame Street! Model Extraction of BERT-based APIs

By Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer. 2019

261 citations recorded by the source corpus.

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, background): “There is an existing body of work in this direction, e.g., one can detect whether a particular model is trained via imitation (Krishna et al., 2020; Juuti et al., 2019; Szyller et al., 2019; Maini et al., 2021) or slow model stealing by sacrifing some performance (Wallace et al., 2020; Orekondy et…”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Such heuristics can even be used to reconstruct a non-publiclyavailable model (Krishna et al., 2020).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “Pioneering work [27, 9, 41] has demonstrated that it is possible to extract private pre-training data from BERT [15], GPT-2 [43], and ChatGPT [1], respectively.”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, background): “For example, watermarking strategies (Zhang et al., 2018; Szyller et al., 2019; Krishna et al., 2020) intentionally output incorrect responses for certain inputs and then tests if the suspected stolen model reproduces the mistakes.”
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations (2020, background): “n be created by copying the response of a black-box model on a relatively small set of exciting inputs, at least in the restricted cases where every model is ne-tuned from a common pre-trained models [21]. While our bounds are tighter that others proposed in the literature, our model has limitations”
Increasing the Cost of Model Extraction with Calibrated Proof of Work (2022, background): “Furthermore, an adversary can extract a functionally similar model even without access to any real input training data (Krishna et al., 2020; Truong et al., 2021; Miura et al., 2021) while bypassing the long and expensive process of data procuring, cleaning, and preprocessing.”
Black-box Prompt Learning for Pre-trained Language Models (2022, background): “Second, the centralizing nature of PLMs exposes them to potential attacks to cause a single-point failure (Krishna et al., 2019).”
Compositional Semantic Parsing with Large Language Models (2022, unknown): “For pretrained models, leakage of the test data is a potential concern (Krishna et al., 2020; Carlini et al., 2021).”
Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models (2024, unknown): “In NLP, transformer-based models built with BERT have also been successfully stolen, with extracted models performing nearly as well as the original API models for tasks such as sentiment classification, question answering, and natural language inference (Krishna et al., 2019).”
JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols (2026, unknown): “PLM – ∼ 340M ✓ × – – – – ✓ Thieves [21] Nat.”