Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

By Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, Joey Gonzalez. 2020

187 citations recorded by the source corpus.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, neutral): “…(Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019; Zhang et al., 2018).”
Sparse is Enough in Scaling Transformers (2021, background): “Model pruning [24, 2] makes matrices smaller by removing unneeded weights after or during training, however, the gains in computational complexity on sparse matrices often do”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “This is because larger PLMs show superior sample efficiency and training efficiency over their smaller counterparts (Kaplan et al., 2020; Li et al., 2020) due to overparameterization (Arora et al.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, background): “As shown in previous works [Li et al., 2020, Han et al., 2016] quantization and pruning are complimentary and can be combined to further improve the performance/size ratio.”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “[101] demonstrated that in some instances, training large models and then compressing the results leads to better performance.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “Larger neural language models are known to be more sample efficient and require fewer optimization steps to reach the same performance [46] while also converging faster [61], where performance is usually defined as test perplexity.”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, result): “Suppose that we can tolerate some reasonable PPL degradation, we notice that larger models tend to be compressed by a higher compression ratio, which is consistent with a report in [35].”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “A recent study by Li et al. (2020c) highlights the importance of ﬁrst training over-parameterized models and then compressing them, instead of directly training smaller models, to reduce the performance errors.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, background): “Unstructured pruning (Frankle & Carbin, 2018; Li et al., 2020; Chen et al., 2020; Sanh et al., 2020) prunes individual neurons instead of structured blocks.”
Should You Mask 15% in Masked Language Modeling? (2022, background): “Larger models also exhibit faster convergence for a given computational budget: Li et al. (2020) suggest it is more efficient to train larger models for fewer steps, as opposed to training smaller models for longer.”
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020, background): “Recent trends show that scaling models up (Liu et al., 2019b; Lan et al., 2019; Raf-fel et al., 2019; Li et al., 2020) in terms of computation still improves end task performance, raising ∗ Equal contribution. questions about whether and how the most accurate models can be applied in real-world…”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “We believe that our work not only offers practical insights for those working on LLM training, but also paves the way for future research in this rapidly evolving field.”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Model pruning and quantization [38, 39, 55, 79] are the dominant model compression approaches.”
Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision (2020, result): “Since some smaller tasks in GLUE are reported as unstable (Dodge et al., 2020), recent papers (e.g., Li et al. (2020b)) only report on selected tasks.”