Why are Adaptive Methods Good for Attention Models?

By Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, Suvrit Sra. 2020

381 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution (2022, background): “These stepsizes are known to work very well in deep learning [35], and include Adam [19], Adagrad [11], and RMSprop [29].”
Adaptive Federated Optimization (2020, background): “…learning community because they can reduce the need to tune the learning rates and often exhibit strong empirical performance especially in presence of heavy-tail noise distributions of stochastic gradients (Reddi et al., 2019; McMahan and Streeter, 2010; Duchi et al., 2011; Zhang et al., 2019).”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, positive): “Zhang et al. (2020); Crawshaw et al. (2022) found out per-coordinate gradient clipping can function as adaptivity.”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, methodology): “When trained on ImageNet (Deng et al., 2009) with the conventional Inception-style data preprocessing (Szegedy et al., 2016), Transformers “yield modest accuracies of a few percentage points below ResNets of comparable size” (Dosovitskiy et al., 2021).”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “Finally, a recent study suggested that gradient clipping can improve the performance of standard SGD for non-private training and thus SGD can potentially be used as a replacement of adaptive optimizers (Zhang et al., 2020).”