Scaling Laws for Transfer

By Danny Hernandez, Jared Kaplan, Jared Kaplan, T. Henighan, Sam McCandlish. 2021

330 citations recorded by the source corpus.

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data"transferred"from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, positive): “Understanding the scaling behaviour of language models and their transfer properties has been important in the development of recent large models [23, 18].”
Unified Scaling Laws for Routed Language Models (2022, background): “…that the performance of models on many interesting classes of problems are well understood as power-laws; where a multiplicative increase in model size leads to an additive reduction in the model’s loss [Kaplan et al., 2020, Hernandez et al., 2021, Henighan et al., 2020, Rosenfeld et al., 2019].”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “Despite the insights offered in scaling laws research (Kaplan et al., 2020; Hernandez et al., 2021) there remain unresolved questions: Should one follow fixed scaling ratios? If not, should one scale by depth? Or by width? Will scaling experiments on upstream pre-training generalize for downstream t”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, background): “Since pretraining data can greatly inﬂuence performance (Hernandez et al., 2021; Gururangan et al., 2020) and since mathematics is a small fraction of online text, we introduce a large and diverse mathematics pretraining corpus.”
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022, background): “Empirical studies on scaling laws for neural language models have shown that model performance improves with increasing model capacity and training data (Kaplan et al., 2020; Hernandez et al., 2021; Hoffmann et al., 2022; Tay et al., 2022).”
Scaling Laws for Reward Model Overoptimization (2022, background): “Scaling laws have seen substantial success in machine learning for predicting properties of language models [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021] and has led to better theoretical understanding of language models [Sharma and Kaplan, 2020, Bahri et al., 2021].”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “…there are regular and predictable patterns in the behavior of trained language models as they scale (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Mikami et al., 2021; Pu et al., 2021; Sharma & Kaplan, 2020; Ghorbani et al., 2021), but prior work connecting these “Scaling…”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML.”
Predictability and Surprise in Large Generative Models (2022, background): “) [35], audition [21], transfer from text to programming [36], few-shot adaptation of vision models [54], and more.”
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers (2021, methodology): “Hernandez et al. (2021) use language models pretrained on text and ﬁne-tuned on python code to show that when the ﬁne-tuning dataset is small, models trained from scratch eventually hit a performance wall no matter the model size, while equivalently sized pre-trained models keep improving.”
Scaling Laws for Neural Machine Translation (2021, background): “For many of these tasks the scaling behavior of neural networks is highly predictable; model ﬁt or test loss can be described precisely as a function of its number of parameters (Hestness et al., 2017; Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Rosenfeld et al., 2019).”
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “The progress in these models’ abilities has been fueled by multiple factors, including increased model parameter count, greater training data volume, and enhanced training conﬁgurations (Brown et al., 2020; Radford et al., 2019; Hernandez et al., 2021; Kaplan et al., 2020).”
The Efficiency Misnomer (2021, background): “Many research works that study the scaling law (Kaplan et al., 2020; Hernandez et al., 2021; Tay et al., 2021b), especially in the NLP domain, use the number of parameters as the primary cost indicator (Devlin et al., 2018; Raffel et al., 2019; Liu et al., 2019; Xue et al., 2021).”
Pretrained Transformers as Universal Computation Engines (2021, background): “(Hernandez et al. 2021) do a thorough investigation of transfer with language pretraining, notably showing transfer from English to Python, which they consider to be reasonably distanced from English; many works have also looked at transferring from one langauge to another (Artetxe, Ruder, and…”
Efficient Training of Language Models to Fill in the Middle (2022, background): “It is possible that there is ossification [Hernandez et al., 2021] in the learned document-wide attention pattern in regular AR pretraining which requires a lengthy finetuning stage to adapt to the attention pattern needed in FIM.”
Emergent and Predictable Memorization in Large Language Models (2023, methodology): “The literature on scaling laws [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021, Mikami et al., 2021, Hoffmann et al., 2022] has been successfully used to inform the decision-making of a variety of researchers at model training-time by allowing them to generalize the decisions…”