On the Transformer Growth for Progressive BERT Training

By Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, Jiawei Han. 2020

61 citations recorded by the source corpus.

As the excessive pre-training cost arouses the need to improve efficiency, considerable efforts have been made to train BERT progressively–start from an inferior but low-cost model and gradually increase the computational complexity. Our objective is to help advance the understanding of such Transformer growth and discover principles that guide progressive training. First, we find that similar to network architecture selection, Transformer growth also favors compound scaling. Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth operators in each dimension via controlled comparison to give practical guidance for operator selection. In light of our analyses, the proposed method CompoundGrow speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively while achieving comparable performances.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Yi Tay
4. Jie Zhou
5. Mostafa Dehghani
6. Qun Liu
7. Yankai Lin
8. Neil Houlsby

In Computer Vision and Pattern Recognition

1. Jie Zhou
2. Maosong Sun
3. Zhiyuan Liu
4. Neil Houlsby
5. Mostafa Dehghani
6. Qun Liu
7. Yi Tay
8. Basil Mustafa

Citations in the corpus

• ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Another line of work proposes progressive training to dynamically expand an existing PLM’s size through parameter recycling (Gong et al., 2019; Gu et al., 2021; Chen et al., 2021).”
• Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, background): “Gu et al. (2020) show that compound scaling (scaling depth, width and input length together) is favorable and propose a strategy with various growing operators on each dimension.”
• bert2BERT: Towards Reusable Pretrained Language Models (2021, background): “Some works (Gong et al., 2019; Yang et al., 2020; Gu et al., 2021) propose progressive learning to accelerate the pre-training, which are motivated by the fact that different layers have some similar knowledge (e.”