Extreme Tensoring for Low-Memory Preconditioning
By Xinyi Chen, Naman Agarwal, Elad Hazan, Cyril Zhang, Yi Zhang. 2019
11 citations recorded by the source corpus.
State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods. We propose extreme tensoring for high-dimensional stochastic optimization, showing that an optimizer needs very little memory to benefit from adaptive preconditioning. Our technique applies to arbitrary models (not necessarily with tensor-shaped parameters), and is accompanied by regret and convergence guarantees, which shed light on the tradeoffs between preconditioner quality and expressivity. On a large-scale NLP model, we reduce the optimizer memory overhead by three orders of magnitude, without degrading performance.
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Luke S. Zettlemoyer
- Tim Dettmers
- Michael Lewis
- Sam Shleifer
- Mohammad Rastegari
- Sachin Mehta
- Torsten Hoefler
- Tal Ben-Nun
Citations in the corpus
- 8-bit Optimizers via Block-wise Quantization (2021, methodology): “AdaGrad that uses only the main diagonal as optimizer state and extensions of AdaGrad such as SM3 (Anil et al., 2019) and extreme tensoring (Chen et al., 2020a) can be more efficient than 8-bit Adam.”