On the Difficulty of Defending Self-Supervised Learning against Model Extraction

By Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot. 2022

36 citations recorded by the source corpus.

Self-Supervised Learning (SSL) is an increasingly popular ML paradigm that trains models to transform complex inputs into representations without relying on explicit labels. These representations encode similarity structures that enable efficient learning of multiple downstream tasks. Recently, ML-as-a-Service providers have commenced offering trained SSL models over inference APIs, which transform user inputs into useful representations for a fee. However, the high cost involved to train these models and their exposure over APIs both make black-box extraction a realistic security threat. We thus explore model stealing attacks against SSL. Unlike traditional model extraction on classifiers that output labels, the victim models here output representations; these representations are of significantly higher dimensionality compared to the low-dimensional prediction scores output by classifiers. We construct several novel attacks and find that approaches that train directly on a victim's stolen representations are query efficient and enable high accuracy for downstream models. We then show that existing defenses against model extraction are inadequate and not easily retrofitted to the specificities of SSL.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Dawn Song
4. Eric Wallace
5. Chelsea Finn
6. Mohit Bansal
7. Percy Liang
8. Trevor Darrell

In Representation & Self-Supervised Learning

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Eric Wallace
5. Xinyang Geng
6. Dawn Song
7. Hao Liu
8. Han-Jia Ye

In Safety, Robustness & Interpretability

1. Eric Wallace
2. Sergey Levine
3. Dawn Song
4. Pieter Abbeel
5. Hao Liu
6. Nicholas Carlini
7. Florian Tramèr
8. Matthias Hein

Citations in the corpus

• The False Promise of Imitating Proprietary LLMs (2023, background): “…this direction, e.g., one can detect whether a particular model is trained via imitation (Krishna et al., 2020; Juuti et al., 2019; Szyller et al., 2019; Maini et al., 2021) or slow model stealing by sacrifing some performance (Wallace et al., 2020; Orekondy et al., 2020; Dziedzic et al., 2022a,b).”