MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

By Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou. 2020

2,387 citations recorded by the source corpus.

Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its variants) have achieved remarkable success in varieties of NLP tasks. However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this work, we present a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation. The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher). Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student. Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys) that have been used in existing works. Moreover, we show that introducing a teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large pre-trained Transformer models. Experimental results demonstrate that our monolingual model outperforms state-of-the-art baselines in different parameter size of student models. In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model. We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Materials Chemistry, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Materials Chemistry

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “MiniLM (Wang et al., 2020) conducts deep self-attention distillation also at pre-training stage.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Related work may include distillation (Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al., 2019; Fan et al., 2019).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
A Primer in BERTology: What We Know About How BERT Works (2020, neutral): “4 93% ×9 BERT8 †‖ SQuAD MiniLM (Wang et al., 2020b) ×1.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “There are some typical works employing knowledge distillation for PTMs, such as DistillBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019), BERT-PKD (Sun et al., 2019b) and MiniLM (Wang et al., 2020d).”
Knowledge Distillation: A Survey (2020, methodology): “To address this problem, several lightweight variations of BERT (called BERT model compression) using knowledge distillation are proposed (Sun et al. 2019; Jiao et al. 2020; Tang et al. 2019; Sanh et al. 2019; Wang et al. 2020a; Liu et al. 2020b; Fu et al. 2021).”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Thus the knowledge that can be distilled in a BERT model is also much richer (Jiao et al., 2019; Wang et al., 2020).”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “We adopt the MiniLM [54] structure to reduce the computation cost of the transformer module after comparing different compact BERT models.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, methodology): “[72] shows that using just the last transformer block’s attention map yields equivalent results, allowing the Teacher and Student to have a different number of layers.”
Taxonomy of Risks posed by Language Models (2022, background): “ciency gains during training and inference [116], for example via pruning [164], distillation [93, 189], or fine-tuning [148].”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, methodology): “We cluster all the queries and embed each query by a popular pre-trained language understanding model all-MiniLML6-v2 [68].”
Pre-trained Summarization Distillation (2020, methodology): “…can be shrunk substantially without much performance degradation on the GLUE suite of non-generative tasks using direct Knowledge Distillation (“KD”) (Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2020; Wang et al., 2020; Devlin et al., 2018; Wang et al., 2018; Hinton et al., 2015).”
Deep Learning--based Text Classification (2020, methodology): “Developing a task-agnostic model compression method is an active research topic [226].”
A Survey on Vision Transformer (2020, background): “A teacher assistant [86] is also introduced in [127], which reduces the gap between the large pre-trained Transformer models and compact student networks to make the mimicking takes more easily.”
MemoryBank: Enhancing Large Language Models with Long-Term Memory (2023, methodology): “In conclusion, these stages transform SiliconFriend from a standard AI chatbot into a long-term AI companion, capable of remembering and learning from past interactions to provide personalized and empathetic user experience.”
How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting (2022, neutral): “On the other hand, as proposed in [45], we exploit the self-attention coefﬁcients A Tp, [”