Adversarial Self-Supervised Data Free Distillation for Text Classification

By Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, Weiming Lu. 2020

27 citations recorded by the source corpus.

Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computationally expensive model to a resource-efficient lightweight model. However, most KD algorithms, especially in NLP, rely on the accessibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two-stage data-free distillation method, named Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed for compressing large-scale transformer-based models (e.g., BERT). To avoid text generation in discrete space, we introduce a Plug & Play Embedding Guessing method to craft pseudo embeddings from the teacher's hidden knowledge. Meanwhile, with a self-supervised module to quantify the student's ability, we adapt the difficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Ricardo Henao
3. Tong Yu
4. Gongfan Fang
5. Xinyin Ma
6. Sungchul Kim
7. Rui Wang
8. Handong Zhao

Citations in the corpus

• Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “AS-DFD is designed for text classification, where they use the feature of the special token [CLS] from BERT for classification, while features of the non-special tokens (within text) are trained with an augmented task of language modeling.”
• LLM-Pruner: On the Structural Pruning of Large Language Models (2023, methodology): “Besides, [31, 30, 39] proposes methods that distill the model without reliance on the training corpus of the model.”