ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning

By Zi-Mei Yao, A. Gholami, Sheng Shen, Kurt Keutzer, Michael W. Mahoney. AAAI Conference on Artificial Intelligence, 2020

383 citations recorded by the source corpus.

Incorporating second-order curvature information into machine learning optimization algorithms can be subtle, and doing so naïvely can lead to high per-iteration costs associated with forming the Hessian and performing the associated linear system solve. To address this, we introduce ADAHESSIAN, a new stochastic optimization algorithm. ADAHESSIAN directly incorporates approximate curvature information from the loss function, and it includes several novel performance-improving features, including: (i) a fast Hutchinson based method to approximate the curvature matrix with low computational overhead; (ii) a spatial averaging to reduce the variance of the second derivative; and (iii) a root-mean-square exponential moving average to smooth out variations of the second-derivative across different iterations. We perform extensive tests on NLP, CV, and recommendation system tasks, and ADAHESSIAN achieves state-of-the-art results. In particular, we find that ADAHESSIAN: (i) outperforms AdamW for transformers by0.13/0.33 BLEU score on IWSLT14/WMT14, 2.7/1.0 PPLon PTB/Wikitext-103; (ii) outperforms AdamW for Squeeze-Bert by 0.41 points on GLUE; (iii) achieves 1.45%/5.55%higher accuracy on ResNet32/ResNet18 on Cifar10/ImageNetas compared to Adam; and (iv) achieves 0.032% better score than Adagrad for DLRM on the Criteo Ad Kaggle dataset. The cost per iteration of ADAHESSIANis comparable to first-order methods, and ADAHESSIAN exhibits improved robustness towards variations in hyperparameter values. The code for ADAHESSIAN is open-sourced and publicly-available [1].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/20438e2a38a0c4723fbd9de50b44b7335f6f43cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, positive): “For any loss function (cid:96) ( θ ) on parameters θ ∈ R d , the Hutchin-son’s estimator (Hutchinson, 1989; Roosta-Khorasani & Ascher, 2015; Yao et al., 2021) ﬁrst draws u ∈ R d from the spherical Gaussian distribution N (0 , I d ) , and then outputs ˆ h = u (cid:12) ( ∇ 2 ( θ ) u ) , where (cid:12)”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “These constraints have inspired significant work on efficient numerical approximations of the Hessian for deep neural networks, such as the line of work on the K-FAC approximation [19, 31, 5, 49], or efficient block-wise approximations [47, 37].”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, background): “Since the first release of PYHESSIAN, several other high-quality implementations of second-order methods have appeared and been applied to NN problems [2, 28].”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “To solve this problem, we adopt the widely used technique (Bishop and Nasrabadi 2006; Xu, Roosta, and Mahoney 2020; Yao et al. 2021) of approximating the Hessian matrix with a diagonal matrix denoted as H = diag{h1, h2, · · · , hn}.”
AI and Memory Wall (2024, positive): “These methods are typically more robust to hyperparameter tuning, and they can achieve SOTA [43].”
A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, neutral): “Many approximation methods can be viewed as diagonal approximations of the curvature matrix (Becker and Lecun, 1988; John et al., 2011; Tieleman and Hinton, 2012;Kingma and Ba, 2014; Yao et al., 2020), and these algorithms are computationally tractable.”