NAMSG: An Efficient Method For Training Neural Networks
By Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Li'an Qiao, Wei Xue, Haohuan Fu, Guangwen Yang. 2019
We introduce NAMSG, an adaptive first-order algorithm for training neural networks. The method is efficient in computation and memory, and is straightforward to implement. It computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating vector elementwise by a nonincreasing preconditioner to take the advantages of AMSGRAD. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, and provide a guideline to select the observation distance without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting. Experiments demonstrate that NAMSG works well in practice and compares favorably to popular adaptive methods, such as ADAM, NADAM, and AMSGRAD.
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Christopher J. Shallue
- Chris J. Maddison
- Dami Choi
- George E. Dahl
- Jaehoon Lee
- Zachary Nado
- Yann LeCun
- Samuel L. Smith
Citations in the corpus
- On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Szegedy et al. (2016) used (cid:15) = 1 for RMSProp; Liu et al. (2019) reported that their results were sensitive to (cid:15) and set (cid:15) = 10 − 6 for Adam; Tan et al. (2019) and Tan and Le (2019) set (cid:15) = 10 − 3 for RMSProp, the latter achieving state-of-the-art ImageNet top-1 accuracy.”