On the Ineffectiveness of Variance Reduced Optimization for Deep Learning

By Aaron Defazio, L. Eon Bottou. 2018

127 citations recorded by the source corpus.

The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, background): “…theoretical results on non-convex minimization (Reddi et al., 2016; Allen-Zhu and Hazan, 2016), the effectiveness of variance reduced methods is still an open question, and a recent technical report by Defazio and Bottou (2018) provides negative empirical results on the variance reduction aspect.”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “The performance of MimeLite can be attributed to the observation that the SVRG correction may not be necessary in deep learning [5, 34].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “in Figure 1, Table 1 and Figure 4(b), as well as in [7, 46].”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “While Katyusha has theoretical guarantees in the convex optimization setting, the SVRG-based update does not work well for neural networks [4].”
Momentum-Based Variance Reduction in Non-Convex SGD (2019, background): “this improvement, SVRG has not seen as much success in practice in non-convex machine learning problems [4].”
Coresets for Data-efficient Training of Machine Learning Models (2019, background): “However, since in deep networks the gradients may change rapidly after a small number of parameter updates (Defazio & Bottou, 2019), selecting smaller subsets and more frequent updates result in a larger improvement over the random baseline.”
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning (2019, methodology): “Our method can be viewed as seeking to remove the ‘client-variance’ in the gradients across the clients, though there still remains additional stochasticity as in (Kulunchakov & Mairal, 2019), which is important in deep learning (Defazio & Bottou, 2019).”
Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (2019, background): “But, at least for now, they seem to have limited applicability in deep learning (Defazio and Bottou, 2018).”
Projection-Free Adaptive Gradients for Large-Scale Optimization (2020, methodology): “Since variance reduction can be ineffective for the training of deep neural networks (Defazio and Bottou, 2019), we run AdaSFW only.”