MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

By Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei. Findings, 2020

413 citations recorded by the source corpus.

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b5b006dc558cb7fbd532d67e989173b536e8ac80

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “We note that this strategy is very different from the deep knowledge distillation strategy used in [48, 53, 49], where knowledge distillation is performed with a single stage that minimizes the sum of the losses from prediction logits, hidden states, and attention maps.”
Language model compression with weighted low-rank factorization (2022, methodology): “The representative works of path-1 include DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019), MobileBERT (Sun et al., 2020), and MiniLM v1/v2 (Wang et al., 2020b;a).”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Pre-training We pre-train on our proposed text pair dataset for three model sizes: E5 small , E5 base and E5 large initialized from MiniLM [59], bert-base-uncased , and bert-large-uncased-whole-word-masking respectively.”
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021, positive): “Without knowledge distillation, DeBERTaV3small outperforms MiniLMv2small [38] by 1.”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “The second problem is that matching attention module [9, 10] in Transformer between teacher and students will leave the weights of feed forward network in the students’ last layer untrained, as one Transformer layer is composed of an attention module and a feed forward network module [67].”
Multilingual E5 Text Embeddings: A Technical Report (2024, positive): “The mE5 small , mE5 base and mE5 large are initialized from the multi-lingual MiniLM (Wang et al., 2021), xlm-roberta-base (Conneau et al., 2020), and xlm-roberta-large respectively.”
A Fair Benchmarking of Deep Relational Database Learning Models (2026, unknown): “For example, DBFormer uses GloVe embeddings [23], Griffin uses Nomic Embed [24], and RT uses MiniLM-v2 [25].”
Granite Embedding Models (2025, unknown): “We hypothesize this being an artifact of our starting point, the 6 layer WatBERT model, as the 6 layer WatBERT model is distilled from the 12 layer WatBERT 4.1 using multi-head self-attention relation distillation (Wang et al., 2021) which does not train the Masked Language Modeling head.”
Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, background): “The average of the final hidden states of only the input sample is calculated during mean pooling, while disregarding the instruction and format tokens.”
Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models (2024, methodology): “We looked to variants of the MiniLMv2 architecture (Wang et al., 2021) for our smaller sizes ( xs and s ), and we opted for the Nomic BERT architecture (Nussbaum et al., 2024) for our long-context variant ( m-long ).”
Open-Source Embedding Models: A Comprehensive Survey of Techniques, Benchmarks, and Applications (2026, unknown): “Developed through a process known as deep self-attention distillation, MiniLM is a distilled version of large transformer-based models like BERT [40], [41].”
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026, unknown): “We therefore add an optional feature-matching term that regularizes selected student features toward teacher features, such as hidden states, layer-wise representations [18], self-attention relations or attention-derived features [19], or other task-relevant internal signals.”