Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

By Iulia Turc, Ming-Wei Chang, Kenton C. T. Lee, Kristina Toutanova. 2019

775 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “It has become common practice to pre-train large models and distill them down to smaller ones (Sun et al., 2019; Turc et al., 2019) for real applications.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “Based on table 5 and Turc et al., 2019’ results, we should prioritize depth over width in auto-encoder models for both better downstream NLU task performance and increasing safety.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “train the full PARADE model using a smaller BERT variant distilled from BERTLarge by Turc et al. [2019] in place of BERTBase, and 2.”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, methodology): “E.3 Larger BERT Models are Better We also compare different sizes of BERT (Turc et al., 2019) (41, 110, and 330M) parameters.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “…et al., 2019; Jiao et al., 2019), mimicking the activation patterns of individual portions of the teacher network (Sun et al., 2019a), and knowledge transfer at different stages at the pre-training (Turc et al., 2019; Jiao et al., 2019; Sun et al.) or at the ﬁne-tuning stage (Jiao et al., 2019)).”
Learning inverse folding from millions of predicted structures (2022, methodology): “For model scaling experiments, we followed the model sizes in (Turc et al., 2019), and chose the 142-million-parameter model with 8 encoder layers, 8 decoder layers, 8 attention heads, and embedding dimension 512 based on the best validation set performance (Figure 6c shows test set ablation).”
Knowledge Distillation: A Survey (2020, methodology): “For a task-speciﬁc distillation (Turc et al. 2019), the performance of knowledge distillation with the interactions among pre-training, distillation and ﬁne-tuning for the compact student model is studied.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “r pretrained models (for subsequent ﬁnetuning) through weight pruning (Gordon et al., 2020; Sajjad et al., 2020; Chen et al., 2020) and/or knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Turc et al., 2019; Jiao et al., 2019; Sun et al., 2020b). It would be interesting to see whether our”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “Turc et al. (2019) propose to pre-train the smaller BERT models to improve task-specific knowledge distillation.”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classiﬁcation [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “DistillBERT [28] 6 66M CEKD + CosKD + CEMLM 3 (unlabeled) 3 PD-BERT [37] 6 66M CEMLM + CEKD + CETASK 3 (unlabeled) 3”
Measuring and Reducing Gendered Correlations in Pre-trained Models (2020, positive): “We investigate if a trend by BERT model size exists, which would make small models particularly susceptible to issues (e.g. from encoding shallow heuristics rather than nuanced associations), by evaluating the new, smaller models from Turc et al. (2019) (Table 3).”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “BERT is presumably more expressive because the mixing (attention) weights are both task speciﬁc and token dependent, determined Turc et al. (2019), for all models, we ﬁx the feed-forward size to 4 d h and the number of self-attention heads to d h / 64 .”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “Our proposed methods are simpler and complementary to distillation as they can improve the pre-training of compact student models (Turc et al., 2019).”
A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “[11] introduce the concept of pretrained distillation to improve the efficiency of knowledge distillation and release several distilled versions of BERT (BERT-small, BERTmini, BERT-tiny).”