When Do You Need Billions of Words of Pretraining Data?

By Yian Zhang, Alex Warstadt, Haau-Sing Li, Samuel R. Bowman. 2020

164 citations recorded by the source corpus.

NLP is currently dominated by language models like RoBERTa which are pretrained on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsupervised relative acceptability judgments, unsupervised language model knowledge probing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Emergent Abilities of Large Language Models (2022, positive): “(Wei et al., 2021b), which has even been shown to have striking “aha” moments similar to those in the psycholinguistics literature (Abend et al., 2017; Zhang et al., 2021).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Concerning the latter, Zhang et al. (2021) found that models require less data to encode syntactic and semantic properties compared to commonsense knowledge.”
Comparing Test Sets with Item Response Theory (2021, methodology): “Figure 3 shows the performance of RoBERTaLarge, ALBERT-XXL-v2, and one of the low performing MiniBERTas (RoBERTa-Med-Small-1M-2) on all validation sets.”
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair (2021, unknown): “We consider a diverse set of pre-199 trained Transformer models: BERT (Devlin et al.,200 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan201 et al., 2020), XLM-R (Conneau et al., 2020), ELEC-202 TRA (Clark et al., 2020), MiniBERTa (Zhang et al.,203 2021), BART (Lewis et al., 2020), and DeBERTa-204 V2 ”
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, background): “Then, PLMs have already encoded language patterns in their parameters after pre-training (Zhang et al. 2021), however, the parameters of GNNs are usually randomized.”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, background): “Zhang et al. (2020) showed that pre-training RoBERTa requires 10M to 100M words to learn representations that reliably encode most syntactic and semantic features.”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, background): “Zhang et al. (2020) show that different tasks exhibit sigmoid-like curves in terms of how much pretraining data is required to achieve good results before performance levels off.”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, methodology): “Pretrained Transformer Models We use 18 Transformer-based models: ALBERT-XXL-v2 (Lan et al., 2020), RoBERTa LARGE and RoBERTa BASE (Liu et al., 2019b), BERT LARGE and BERT BASE (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and 12 MiniBERTas (Zhang et al., 2021).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “Unlike prior work, which typically lacks either language (Rönnqvist et al., 2019; Zhang et al., 2020) or task diversity (Wu and Dredze, 2020; Vulić et al.”
When Combating Hype, Proceed with Caution (2021, background): “Better Forecasting Scaling laws results in NLP (Hestness et al., 2017; Kaplan et al., 2020; Brown et al., 2020; Zhang et al., 2021) offer the promise that we can predict the performance of future larger-scale machine learning models on at least some metrics.”
Changing the World by Changing the Data (2021, background): “For example, Zhang et al. (2020) compared the learning rates for different linguistic phenomena as RoBERTa was pre-trained on more and more data.”
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “Prior work has shown that the predictions of neural LMs have high quality with respect to syntax (Hu et al., 2020; Zhang et al., 2021b); e.g., the annotators in Dou et al. (2022) marked less than 3% of LM-generated tokens as having grammatical errors.”