Poor Man's BERT: Smaller and Faster Transformer Models

By Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Preslav Nakov. 2020

131 citations recorded by the source corpus.

Subfields: Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “We demonstrate the effectiveness of TinyBERT by including more baselines such as Poor Man’s BERT (Sajjad et al., 2020), BERT-of-Theseus (Xu et al.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “[33] show that top-layer dropping consistently outperforms other strategies when dropping 6 layers, so we drop top 6 layers from BERTBASE for truncated BERTBASE.”
Of Non-Linearity and Commutativity in BERT (2021, background): “Similarly, in [23] the authors exploit layer similarity to re-order the layers and improve performance, while in [7] and [29] they analyze layer redundancy to prune BERT with minimal performance degradation.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “Code: https://github.com/dguo98/DiffPruning A popular approach to parameter-efficiency is to learn smaller compressed models for each task (Gordon et al., 2020; Sajjad et al., 2020; Zhao et al., 2020; Sanh et al., 2020).”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, background): “Top-layer dropping has been proven to be a strong baseline (Sajjad et al., 2020).”
Learned Token Pruning for Transformers (2021, methodology): “[14, 37] use LayerDrop to train a model that will have adaptive depth during training and drop unimportant layers during fine-tuning.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, methodology): “Next, we compare ensembled results to better demonstrate memory efﬁciency of the masking 5 Similar observations were made: DistilBERT has a 10% accuracy drop on RTE compared to BERT-base (Sanh et al., 2019); Sajjad et al. (2020) 6 Discussion”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “This also showcases that XLNet is a much better candidate for model compression where several higher layers can be pruned with marginal loss in performance, as shown by Sajjad et al. (2020).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[166] proposed different strategies to drop encoder layers in pretrained language models.”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, background): “[66] investigated different layer pruning strategies and iden-”
Intriguing Properties of Compression on Multilingual Models (2022, background): “Model We use the cased multilingual BERT (mBERT) (Devlin et al., 2019) for all our experiments because it is one of the most widely used and studied multilingual LMs (e.g., Pires et al., 2019; Rönnqvist et al., 2019; Wu and Dredze, 2019; Wang et al., 2020; Chi et al., 2020 trained on Wikipedia data ”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, methodology): “For structured pruning, [11, 34] uses LayerDrop to train the model and observes that small/eﬃcient models can be extracted from the pre-trained model; [46] uses a low-rank factorization of the weight matrix and adaptively removes rank-1 components during training; and [27] tests head drop for…”
Block Pruning For Faster Transformers (2021, background): “In contrast, structured pruning removes coherent groups of weights (Murray and Chiang, 2015; See et al., 2016; Joulin et al., 2016; Fan et al., 2020; Sajjad et al., 2020).”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, background): “Note that layer reduction without pretraining has also been addressed in [41].”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “A common hyper-parameter in softmax calculation is the temperature , which controls the smoothness of the output by scaling the logits before applying softmax.”