MobileBERT: Task-Agnostic Compression of BERT by Progressive Knowledge Transfer

By Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou. 2019

20 citations recorded by the source corpus.

Subfields: Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Furu Wei
2. Li Dong
3. Canwen Xu
4. Shaohan Huang
5. Hangbo Bao
6. Wenhui Wang
7. Ming Zhou
8. Wangchunshu Zhou

In Efficiency & Compression

1. Canwen Xu
2. Furu Wei
3. Tao Ge
4. Kurt Keutzer
5. Wangchunshu Zhou
6. Forrest N. Iandola
7. Xipeng Qiu
8. Hangbo Bao

Citations in the corpus

• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “TinyBERT [15] 4 15M MSEattn + MSEhidn + MSEembd + CEKD 3 (unlabeled + labeled) 7 (Transformer only) MobileBERT [34] 24 25M FMT+AT+PKT+CEKD+CEMLM 3 (unlabeled) 7 (BERT only) BERT-of-Theseus (Ours) 6 66M CETASK 7 3”
• Pre-trained models for natural language processing: A survey (2020, methodology): “MobileBERT [159] IB-BERTLARGE Layer-to-Layer distillation with soft target probabilities, hidden states, and selfattention distributions.”
• SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, result): “This is slightly faster than the 620ms reported by the MobileBERT authors in the same setting [42].”
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, methodology): “(Sun et al., 2019b), the reported results are directly fine-tuned on downstream tasks.”
• Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “MOBILEBERT employs inverted bottleneck and bottleneck modules for teacher and student to make their hidden dimensions the same.”