One Epoch Is All You Need

By Aran Komatsuzaki. 2019

64 citations recorded by the source corpus.

In unsupervised learning, collecting more data is not always a costly process unlike the training. For example, it is not hard to enlarge the 40GB WebText used for training GPT-2 by modifying its sampling methodology considering how many webpages there are in the Internet. On the other hand, given that training on this dataset already costs tens of thousands of dollars, training on a larger dataset naively is not cost-wise feasible. In this paper, we suggest to train on a larger dataset for only one epoch unlike the current practice, in which the unsupervised models are trained for from tens to hundreds of epochs. Furthermore, we suggest to adjust the model size and the number of iterations to be performed appropriately. We show that the performance of Transformer language model becomes dramatically improved in this way, especially if the original number of epochs is greater. For example, by replacing the training for 10 epochs with the one epoch training, this translates to 1.9-3.3x speedup in wall-clock time in our settings and more if the original number of epochs is greater. Under one epoch training, no overfitting occurs, and regularization method does nothing but slows down the training. Also, the curve of test loss over iterations follows power-law extensively. We compare the wall-clock time of the training of models with different parameter budget under one epoch training, and we show that size/iteration adjustment based on our proposed heuristics leads to 1-2.7x speedup in our cases. With the two methods combined, we achieve 3.3-5.1x speedup. Finally, we speculate various implications of one epoch training and size/iteration adjustment. In particular, based on our analysis we believe that we can reduce the cost to train the state-of-the-art models as BERT and GPT-2 dramatically, maybe even by the factor of 10.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Scaling Laws for Neural Language Models (2020, neutral): “There are some parallels between our ﬁndings on optimal allocation of compute and [Kom19], including power-law learning curves.”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “As we increase model and dataset sizes, optimization becomes increasingly efﬁcient, until eventually learning curves begin to merge with the L ( D ) trend, so that there are no beneﬁts to be gained from training for more than a single epoch [Kom19].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Such settings are used when training large image models [16, 53], multimodal models [63] and language models [48, 14, 32, 84], the latter of which have come under signiﬁcant scrutiny due to privacy concerns [12, 4].”
Scaling Data-Constrained Language Models (2023, background): “Using multiple epochs is, of course, standard in machine learning generally; however, most prior large language models have been trained for a single epoch [51, 15] and some work explicitly advocates against reusing data [40].”
KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations (2020, methodology): “In the one-epoch training session, we trained the architecture with 1 epoch (Komatsuzaki, 2019) to avoid overfitting and to guarantee the possibility of having a relatively light computational burden.”