Memory-Efficient Adaptive Optimization for Large-Scale Learning

By Rohan Anil, Vineet Gupta, Tomer Koren, Yoram Singer. 2019

50 citations recorded by the source corpus.

Adaptive gradient-based optimizers such as AdaGrad and Adam are among the methods of choice in modern machine learning. These methods maintain second-order statistics of each parameter, thus doubling the memory footprint of the optimizer. In behemoth-size applications, this memory overhead restricts the size of the model being used as well as the number of examples in a mini-batch. We describe a novel, simple, and flexible adaptive optimization method with sublinear memory cost that retains the benefits of per-parameter adaptivity while allowing for larger models and mini-batches. We give convergence guarantees for our method and demonstrate its effectiveness in training very large deep models.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “, 2019c], and even simply as a giant text corpus for testing optimizers [Anil et al., 2019].”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “We also experimented with explicit memory-saving optimizers including SM3 (Anil et al., 2019), Adafactor (Shazeer & Stern, 2018), and NovoGrad (Ginsburg et al., 2019) with mixed results.”
ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, background): “3) Memory Efficient Optimizer: [21], [22] focus on reducing memory consumption of adaptive optimization methods by maintaining coarser-grained statistics of model parameters and gradients, with potential impact on model convergence guarantees.”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “AdaGrad that uses only the main diagonal as optimizer state and extensions of AdaGrad such as SM3 (Anil et al., 2019) and extreme tensoring (Chen et al.”
PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS (2021, methodology): “In all the experiments, the PnG BERT model used 6 Transformer layers with hidden size 512 and 8 attention heads, pretrained using the SM3 optimizer [31] with a batch size of 24K for 1M steps and a maximum sequence length of 480 tokens.”
Momentum Improves Normalized SGD (2020, methodology): “Adam is typically used to achieve state of the art results on BERT task Devlin et al. (2019) whereas SGD is used for Resnet-50 on Imagenet dataset since Adam fails to perform well on it You et al. (2019); Anil et al. (2019).”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “To reduce optimizer memory specifically, space-efficient alternatives to Adam and other adaptive optimizers have recently been proposed, such as Adafactor (Shazeer and Stern, 2018) and SM3 (Anil et al., 2019).”
Big Bidirectional Insertion Representations for Documents (2019, methodology): “All models were trained with the SM3 optimizer (Anil et al., 2019) with momentum 0.”
Large-Scale Differentially Private BERT (2021, background): “…that exploit increased parallelism: such as online distillation (Anil et al., 2018) and ones that use lower memory (Shazeer and Stern, 2018; Anil et al., 2019), automatic tuning of hyper-parameters meta optimization (Amid et al., 2020), more efﬁcient architectures such as MLP-Mixer, and…”