Why ADAM Beats SGD for Attention Models

By Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Surinder Kumar, Suvrit Sra. 2019

88 citations recorded by the source corpus.

While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models. The settings under which SGD performs poorly in comparison to Adam are not well understood yet. In this paper, we provide empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is a root cause of SGD's poor performance. Based on this observation, we study clipped variants of SGD that circumvent this issue; we then analyze their convergence under heavy-tailed noise. Furthermore, we develop a new adaptive coordinate-wise clipping algorithm (ACClip) tailored to such settings. Subsequently, we show how adaptive methods like Adam can be viewed through the lens of clipping, which helps us explain Adam's strong performance under heavy-tail noise settings. Finally, we show that the proposed ACClip outperforms Adam for both BERT pretraining and finetuning tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Further, algorithmic innovations such as momentum [30, 4], adaptivity [17, 41, 42], and clipping [38, 39, 43] are critical for success in deep learning applications and need to be incorporated into the client updates, replacing the SGD update of FEDAVG.”
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, background): “Since CNN and Transformer favor different kinds of optimizer [43], our fully Transformer structure makes it easier to train and fine-tune.”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, result): “Similar observations about SGD were also reported in [72].”
Learning from History for Byzantine Robust Optimization (2020, methodology): “The Gaussian attacker sends to the server vectors of Gaussian distribution of standard deviation 10 8 .”
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “Since the visual backbone and Transformer favor different kinds of optimizers [47], we follow the suggestion of Zhang et al.”
Parameter Prediction for Unseen Deep Architectures (2021, methodology): “Comparing SGD to Adam, the latter performs worse in general except for the ViT architectures similar to [72, 20].”
Understanding the Difficulty of Training Transformers (2020, methodology): “We compare our methods with three initialization methods, i.e. , ReZero (Bachlechner et al., 2020), FixUp (Zhang et al., 2019a), and LookLinear (Bal-duzzi et al., 2017a).”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, unknown): “[68] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “As claimed in [40], the CNN visual backbone and Transformer may favor diﬀerent kinds of the optimizer, we adopt diﬀerent optimizer settings for visual backbone and Transformer.”