Efficient Training of BERT by Progressively Stacking

By Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, Tie-Yan Liu. 2019

192 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2020), model structures/sparsities (Louizos et al., 2018; Gong et al., 2019; Tan & Le, 2019), backpropagation storage requirements (Gruslys et al.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “Networks with 3 or more layers are trained by fine-tuning using the parameters from the depth before (e.g., the 12-layer network parameters are fine-tuned from the checkpoint of the 6-layer network parameters).5 Similar technique has been used in Gong et al. (2019).”
ReZero is All You Need: Fast Convergence at Large Depth (2020, background): “These patterns may hint towards some form of curriculum learning and allow for progressive stacking of layers to further accelerate training [38].”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Gong et al. (2019) note that, since self-attention patterns in higher layers resemble the ones in lower layers, the model training can be done in a recursive manner, where the shallower version is trained ﬁrst and then the trained parameters are copied to deeper layers.”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, methodology): “For depth expansion, previous works generally resort to stacking all the original PLM layers into 2× layers through parameter replication (Gong et al., 2019).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “For those conventional PTMs based on supervised pre-training, we refer to the papers of He et al. (2019) and Zoph et al. (2020).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Other techniques are also available such as progressive training (Gong et al., 2019), or improving initialization (Zhang et al.”
Rethinking Positional Encoding in Language Pre-training (2020, background): “For example, many visualizations (Clark et al., 2019a; Gong et al., 2019) show that the attention distributions of some heads concentrate locally.”
Taking Notes on the Fly Helps BERT Pre-training (2020, unknown): “Gong et al. (2019) observes that parameters in different layers have similar attention distribution, and propose a parameter distillation method from shallow layers to deep layers.”
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping (2020, methodology): “Prior work [31] proposes to accelerate pre-training by first training a 3-layer BERT model and then growing the network depth to 6-layer and subsequently 12-layer.”
Language-agnostic BERT Sentence Embedding (2020, methodology): “For an L layer transformer encoder, we train using a 3 stage progressive stacking algorithm (Gong et al., 2019), where we first learn a L4 layers model and then L 2 layers and finally all L layers.”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “A similar pretraining scheme to the Transformer as we do has been done in [47].”
Variance-reduced Language Pretraining via a Mask Proposal Network (2020, methodology): “[6] observes that parameters in different layers have structural similarity, and the training time of BERT can be reduced using parameter sharing.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “More methods have been introduced to enhance the efficiency of training LMs, such as dynamic architectures (Gong et al., 2019; Zhang & He, 2020) and efficient optimizers (Chen et al.”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, background): “In an effort to reduce total training cost, several works explore progressively growing models during training (Gong et al., 2019; Dong et al., 2020; Li et al., 2020; Shen et al., 2022).”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “A novel method is proposed in order to accelerate BERT training [147].”
BERT Representations for Video Question Answering (2020, background): “BERT is a powerful bidirectional network based on language Trans-formers [39] and it has been shown to outperform LSTMs in several natural language processing tasks [7, 9, 24].”
bert2BERT: Towards Reusable Pretrained Language Models (2021, methodology): “These works (Gong et al., 2019; Yang et al., 2020; Gu et al., 2021) are motivated by the fact that different layers have some similar knowledge (e.”