Adaptive Methods for Nonconvex Optimization

By Manzil Zaheer, Sashank J. Reddi, Devendra Singh Sachan, Satyen Kale, Sanjiv Kumar. 2018

496 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Zaheer et al. (2018) and Liu et al. (2019) consider (cid:15) values orders of magnitude larger than the standard default.”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “More recently, to reduce hand-tuning of hyperparameters, adaptive learning rates for large batch training garnered significant interests (Reddi et al., 2018; Zaheer et al., 2018; Zhang et al., 2019).”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Further, algorithmic innovations such as momentum [30, 4], adaptivity [17, 41, 42], and clipping [38, 39, 43] are critical for success in deep learning applications and need to be incorporated into the client updates, replacing the SGD update of FEDAVG.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “We use the version of Yogi proposed by Zaheer et al. (2018), with first momentum term of β1 = 0.9, second momentum parameter of β2 = 0.99, an initial accumulator value of 0, and an value of 10−5.”
Adaptive Federated Optimization (2020, methodology): “For FEDYOGI, we set the initial accumulator according to the strategy outlined by Zaheer et al. (2018): we use the average of the squares of the coordinates of the gradient, where the gradient is evaluated at the initial model, averaged over multiple batches.”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, neutral): “Methods performing updates only based on the sign of each coordinate of the gradient have recently gaining popularity for training deep learning models (Seide et al., 2014; Carlson et al., 2015; Wen et al., 2017; Balles & Hennig, 2018; Bernstein et al., 2018; Zaheer et al., 2018; Liu et al., 2018).”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “To optimize both variational objectives (5) and (8), we use a mini-batch size of 512 and 3 samples from variational distribution with the Yogi optimizer (Zaheer et al., 2018).”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, background): “, 2018], (4) AdamW [Loshchilov and Hutter, 2019] (5) Yogi [Zaheer et al., 2018] and (6) AdaBound [Luo et al.”
A Sufficient Condition for Convergences of Adam and RMSProp (2018, background): “[23] have established O(1/ √ T ) convergence rate of original Adam directly in the non-convex stochastic setting by requiring the batch size to be the same order as the number of maximum iterations.”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, unknown): “Besides, some previous analyses [13], [14], [70], [72] need the momentum coefﬁcient (i.e., β s) to be close or increased to one, which contradicts with the practice that β s are close to zero.”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, methodology): “L-smoothness may be inappropriate when analyzing other optimization algorithms too It is common for optimization papers seemingly motivated by deep learning to analyze algorithms under the “non-convex but L-smooth”’ setting (Reddi et al., 2016; Agarwal et al., 2017; Zaheer et al., 2018; Zhou et al.,”
SpeedySpeech: Efficient Neural Speech Synthesis (2020, unknown): “We experimented with inverse square root decay and reduce-on-plateau [16] schedules and settled for the latter and a base learning rate 0.”
Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization (2019, background): “• We show a similar convergence result to Zaheer et al. (2018, Thm. 1) for nonconvex stochastic optimization up to the limit of the variance of stochastic gradients under an almost surely bounded step size.”
Generating Accurate Pseudo-Labels in Semi-Supervised Learning and Avoiding Overconfident Predictions via Hermite Polynomial Activations (2019, methodology): “We replicate the experiments re-ported in [29] for the MNIST dataset: we use 4 Hermite polynomials as an activation instead of the original sigmoid.”
Dive into Deep Learning (2020, background): “In a follow-up work (Zaheer et al., 2018) proposed a hotfix to Adam, called Yogi which addresses these issues.”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “Many other hand-designed optimizer architectures have been proposed (Ruder, 2016; Zeiler, 2012; Reddi et al., 2018; You et al., 2019; Liu et al., 2019), but their practical benefits are small in most situations (Schmidt et al.”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, methodology): “• Adam, Yogi, RAdam, MSVAG, AdaBound: We search for optimal β 1 among { 0 .”
Training Neural Networks for and by Interpolation (2019, methodology): “, 2019), we also provide the performance of L4Adam, L4Mom, AdamW (Loshchilov & Hutter, 2019) and Yogi (Zaheer et al., 2018).”