Stochastic Gradient Descent

By Samuel L. Smith, Quoc V. Le. 2014

607 citations recorded by the source corpus.

minimiert wird, wobei λ ein Parameter ist. Um die Notation einfach zu halten, fordern wir im Folgenden u = 0. Dies ist mehr oder weniger ohne Beschränkung der Allgemeinheit, wenn wir u als die d + 1-te Komponente von w interpretieren und an alle xi als letzte Komponente 1 anfügen. Zu einem anderen Zeitpunkt werden wir diese Anspekte noch genauer diskutieren. Führen wir an dieser Stelle etwas Notation ein. Definiere nun `(hw, zi) = max{0, 1− yi〈w,xi〉} ,

Subfields: Finance, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Finance

In Statistics and Probability

Citations in the corpus

Wireless Network Intelligence at the Edge (2018, background): “For a fixed learning rate η, a recent work [197] shows that the noise amount maximizing the test accuracy in centralized ML is characterized by the optimal noise scale g = η(n/B − 1) of the SGLD’s noise εt (see Section IV-A), for n training samples and batch size B.”
A Mean Field Theory of Batch Normalization (2019, background): “This removes subtle gradient fluctuation effects noted in Smith & Le (2018).”
On the linear convergence of the stochastic gradient method with constant step-size (2017, background): “Further connections to the randomized Kaczmarz algorithm can be found in [8] where the case ∩i=1Ci = ∅ is investigated.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “Dziugaite and Roy [7] and Smith and Le [44] provide complementary perspectives on Zhang et al.”
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation (2020, neutral): “There are several choices that one can use including constant step-size [33, 35, 11, 36, 41], decreasing step-size [47, 8, 11, 37, 17] and adaptive step-size [6, 26, 20, 3, 55, 56].”
SmoothOut: Smoothing Out Sharp Minima for Generalization in Large-Batch Deep Learning (2018, background): “Alternative explanations are based on Bayesian learning [19][20].”
Sharpness-aware Quantization for Deep Neural Networks (2021, background): “Following that, several studies have been proposed to investigate the relation between the geometry of the loss landscape and the generalization performance of the models [6, 18, 27, 29, 42, 52, 63].”
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019, methodology): “Why SGD noise enhances final performance remains a mystery (Keskar et al., 2016; Sagun et al., 2017; Mandt et al., 2017; Chaudhari & Soatto, 2017; Smith & Le, 2017).”
Self-adaptive loss balanced Physics-informed neural networks (2021, methodology): “Similarly, the objective is to ﬁnd the best model weights θ ∗ and noise scalar ε ∗ by minimizing the loss L ( ε ; θ ; N ) with gradient-based optimizers, such as Stochastic Gradient Descent (SGD), Adam, and L-BFGS [33,34,35].”
Communication trade-offs for Local-SGD with large step size (2019, methodology): “[25], combining the approach of Bach and Moulines [27] with the correct upper bound for smooth strongly convex SGD [47], and controlling similarly higher order moments.”
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours (2018, background): “that learning rates could be scaled linearly as batch size increases without a noticeable loss in model accuracy [38].”
Optimal SGD Hyperparameters for Fully Connected Networks (2018, background): “It was demonstrated in [11] that the macroscopic behavior of stochastic optimization is governed by a combination g of the introduced hyperparameters, where”
Variance Networks: When Expectation Does Not Meet Your Expectations (2018, background): “Several existing stochastic training procedures have been reinterpreted as special cases of particular Bayesian models, including, but not limited to different versions of dropout [6] drop-connect [15] and even the stochastic gradient descent itself [28].”
Variable-Shot Adaptation for Online Meta-Learning (2020, background): “It is well known in standard supervised learning that the optimal learning rate scales with batch size (Smith & Le, 2018).”
Deep Learning Through the Lens of Example Difficulty (2021, background): “, 2017) and mean-field studies of stochastic optimization (Smith et al., 2021; Stephan et al., 2017; Smith and Le, 2018).”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “However, there is a tradeoff from the viewpoint of generalization; the intrinsic noise stemming from mini-batch gradients provides regularization effects (Chaudhari and Soatto, 2017; Smith and Le, 2017) and by increasing batchsizes, we lose such generalization benefits.”
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects (2018, neutral): “And the work (Smith & Le, 2018) attempted to provide an understanding based on model evidence.”