XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

By Subhabrata Mukherjee, Ahmed Hassan Awadallah. 2020

65 citations recorded by the source corpus.

Deep and large pre-trained language models are the state-of-the-art for various natural language processing tasks. However, the huge size of these models could be a deterrent to using them in practice. Some recent works use knowledge distillation to compress these huge models into shallow ones. In this work we study knowledge distillation with a focus on multilingual Named Entity Recognition (NER). In particular, we study several distillation strategies and propose a stage-wise optimization scheme leveraging teacher internal representations, that is agnostic of teacher architecture, and show that it outperforms strategies employed in prior works. Additionally, we investigate the role of several factors like the amount of unlabeled data, annotation resources, model architecture and inference latency to name a few. We show that our approach leads to massive compression of teacher models like mBERT by upto 35x in terms of parameters and 51x in terms of latency for batch inference while retaining 95% of its F1-score for NER over 41 languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Jianfeng Gao
3. Li Dong
4. Mohit Bansal
5. Erik Cambria
6. Dacheng Tao
7. Wenhui Wang
8. Shaohan Huang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Furu Wei
3. Jianfeng Gao
4. Luc Van Gool
5. Li Dong
6. Yunhe Wang
7. Kai Han
8. Trevor Darrell

In Natural Language Processing

1. Furu Wei
2. Li Dong
3. Jianfeng Gao
4. Ming Zhou
5. Wenhui Wang
6. Graham Neubig
7. Mohit Bansal
8. Luke S. Zettlemoyer

In Efficiency & Compression

1. Manish Gupta
2. Puneet Agrawal
3. Furu Wei
4. Wenhui Wang
5. Li Dong
6. Yunhe Wang
7. Hangbo Bao
8. Kai Han

Citations in the corpus

• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “There have been some works that task-specifically distill pre-trained large LMs into small models [39, 41, 37, 1, 47, 27].”
• Rethinking embedding coupling in pre-trained language models (2020, methodology): “However, distilling a pre-trained model is expensive (Sanh et al., 2019) and requires overcoming architecture differences and balancing training data and loss terms (Mukherjee & Awadallah, 2020).”
• Compression of Deep Learning Models for Text: A Survey (2020, methodology): “[83] also distill a multi-lingual BERT (mBERT) model to a BiLSTM.”
• A Survey on Vision Transformer (2020, methodology): “[88] use the pre-trained BERT [29] as the teacher to guide the training of small models, with the help of large Figure 9.”
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023, background): “Prior works [15, 16] on open-box model show that richer signals such as logits, intermediate representations and attention states can significantly improve distillation performance.”
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, methodology): “It can also be utilized to initialize task-specific distillation (Sun et al., 2019a; Turc et al., 2019; Aguilar et al., 2019; Mukherjee and Awadallah, 2020; Xu et al., 2020; Hou et al., 2020; Li et al., 2020), which uses a finetuned teacher model to guide the training of the student on specific task”
• Deep Learning--based Text Classification (2020, background): “, knowledge distillation [178, 179] to meet the latency and capacity constraints in real-world applications.”
• Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Mukherjee and Awadallah [21] propose a stage-wise scheme to transfer teacher’s intermediate representations.”