A Theory on Adam Instability in Large-Scale Machine Learning

By Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang and 6 more. 2023

56 citations recorded by the source corpus.

We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, background): “Moreover, we direct the reader’s attention to the concurrent work of [41].”
Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “Inspired by [52], we set ϵ = 1 .”