Training Trajectories of Language Models Across Scales

By Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke S. Zettlemoyer, Ves Stoyanov. 2022

81 citations recorded by the source corpus.

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al., 2022)—from 125M to 175B parameters—on next-token prediction, sequence-level generation and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior (Nakkiran et al., 2020); 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; and 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “OPT (Zhang et al., 2022) While more checkpoints of OPT models exist (as is seen by their use in Xia et al. (2022)) they largely are not publicly available (less than 10 checkpoints available, only for the 2.7b, 6.7b, and 13b parameter models).”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “While perplexity has been shown to be a stable and robust metric for language models [68], we also evaluate their zero-shot ability.”
Scaling Data-Constrained Language Models (2023, background): “As loss can be an imperfect proxy for performance on natural language tasks [120, 94, 102], they also validate their conclusions on various downstream tasks.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, unknown): “We compare our LLM-shearing method to other pruning approaches and report validation perplexity, which serves as a strong indicator of overall model capabilities (Xia et al., 2023).”