Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

By Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, J. Demmel, Cho-Jui Hsieh. 2019

126 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “All the model updates use a batch size of 4096 and a LAMB optimizer with learning rate 0.00176 (You et al., 2019).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “Further research (Goyal et al., 2017; You et al., 2017; 2019) has developed techniques to mitigate these effects and drive down the training time of large neural networks.”
ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, methodology): “Adaptive optimization methods [23], [6], [24], [25] are crucial to achieving SOTA performance and accuracy for effective model training of large models.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “You et al. (2019) report that with a batch size of 32k BERT’s training time can be sig-niﬁcantly reduced with no degradation in performance.”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, methodology): “In this section, we only train each model for 125k steps with 2048 batch size, which halves the training schedule of original BERT (Devlin et al., 2018; You et al., 2019).”
Momentum Improves Normalized SGD (2020, methodology): “Adam is typically used to achieve state of the art results on BERT task Devlin et al. (2019) whereas SGD is used for Resnet-50 on Imagenet dataset since Adam fails to perform well on it You et al. (2019); Anil et al. (2019).”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, background): “They propose swapping NSP for a Sentence-Order Prediction objective (SOP), which they suggest better models inter-sentence coherence.”
Sparse-MLP: A Fully-MLP Architecture with Conditional Computation (2021, methodology): “We select a batch size as 4096 at the pre-training stage, LAMB optimizer (You et al. 2019) with weight decay.”
Pre-Training with Whole Word Masking for Chinese BERT (2019, methodology): “Instead of using original AdamWeightDecayOptimizer in BERT, we use LAMB optimizer (You et al., 2019) for better scalability in large batch.”
Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces (2020, methodology): “We use LAMB optimizer (You et al., 2019) to pre-train our model on NVIDIA 3090 GPUs using fairseq toolkit (Ott et al., 2019).”
fastai: A Layered API for Deep Learning (2020, methodology): “For instance, here is the code in fastai, and the algorithm from the paper, for the LAMB optimizer [39]:”
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping (2020, methodology): “Subsequently, Yang et al. propose a layer-wise adaptive large batch optimizer called LAMB [23], allowing to train BERT with 32K batch size on 1024 TPU chips.”