MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

By Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou. 2020

1,053 citations recorded by the source corpus.

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, neutral): “…(Roy et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Sukhbaatar et al., 2019; Child et al., 2019), distillation (Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a;…”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “(1) Smaller and faster models that are either distilled or trained from scratch (Sanh et al., 2019; Sun et al., 2020; Bai et al., 2021; Wang et al., 2020).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Sparse is Enough in Scaling Transformers (2021, background): “Several natural language models used for mobile inference [13, 39] rely on distillation [32] to speed up inference from the pretrained large models.”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “We also exclude MobileBERT (Sun et al., 2020), due to its redesigned Transformer block and different model size.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “Some of the complementary efforts in this area are compact and fast architectures by design (Sun et al., 2020; Iandola et al., 2020), weight sharing (Dehghani et al., 2018; Lan et al., 2019), parameter reuse across multiple downstream tasks (Houlsby et al., 2019; Stickland and Murray, 2019),…”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “For the transformer model, we choose the MiniLM[54] structure after empirically evaluating the performance of several structures on multi-modal representation learning, including BERT [5] and its compact variants [40, 45, 15, 54].”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, methodology): “For example, TinyBERT must employ a uniform-function mapping to selectively choose a subset of the layers for learning, and MobileBERT requires the Teacher and Student to have identical number of layers.”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “Pruning approaches remove parts of the model, typically attention heads (Michel et al., 2019; Voita et al., 2019) while distillation approaches distill a large pre-trained model into a smaller one (Sun et al., 2020).”
Domain-specific Genetic Algorithm for Multi-tenant DNNAccelerator Scheduling (2021, neutral): “Lang ALBERT [59], Bart [62], BERT [24], CamemBERT [68], CTRL [52], DistilBERT [91], ELECTRA [18], FlauBERT [60], GPT2 [82], GPT [81], LongFormer [6], MarianNMT [49], MobileBERT [103], Reformer [53], RetriBERT [44], RoBERTa [64], T5 [83], TransformerXL [22], XLM [58], XLM-RoBERTa [19], XLNet [119] Re”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, neutral): “Sun et al. [181] propose MobileBERT to compress and accelerate the BERT model.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, positive): “As in ImageNet1k experiments, the 4/4 bit configuration delivers better qualities than the 1-bit configuration on all four BERT models, and especially performs well for the hard-to-compress MobileBERT.”