Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping

By Minjia Zhang, Yuxiong He. 2020

124 citations recorded by the source corpus.

Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains. However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses. Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling. In this work, we propose a method based on progressive layer dropping that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency. Extensive experiments on BERT show that the proposed method achieves a 24% time reduction on average per sample and allows the pre-training to be 2.5 times faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Maosong Sun
4. Xipeng Qiu
5. Jie Tang
6. Ji-rong Wen
7. Wayne Xin Zhao
8. Xu Han

In Computer Vision and Pattern Recognition

1. Zhiyuan Liu
2. Qun Liu
3. Minlie Huang
4. Yang Liu
5. Jie Tang
6. Xu Han
7. Xin Jiang
8. Ji-rong Wen

Citations in the corpus

• ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, neutral): “tectures (Zhang and He, 2020), optimization algorithms (You et al.”
• Pre-Trained Models: Past, Present and Future (2021, background): “Some layers can also be dropped during training to reduce the complexity of back-propagation and weight update (Zhang and He, 2020).”
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “More methods have been introduced to enhance the efficiency of training LMs, such as dynamic architectures (Gong et al., 2019; Zhang & He, 2020) and efficient optimizers (Chen et al., 2023; Liu et al., 2023).”
• bert2BERT: Towards Reusable Pretrained Language Models (2021, background): “, 2020), layer dropping training technique (Zhang and He, 2020), etc.”
• OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “Previous work [49] demonstrates that training with Pre-LN BERT avoids vanishing gradients when using aggressive learning rates.”
• Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, unknown): “Notably, one of the promising methods – reducing model sizes via knowledge distillation – has long been studied and shown good performances [13, 60, 42, 53, 49].”