An overview of gradient descent optimization algorithms
By Sebastian Ruder. 2016
6,991 citations recorded by the source corpus.
The statistical testing technique is considered to compare the metrics values of machine learning models on a test set. Since the values of metrics depend not only on the models, but also on the data, it may turn out that different models are the best on different test sets. For this reason, the traditional approach to comparing the values of metrics on a test set is often not enough. Sometimes a statistical comparison of the results obtained on the basis of cross-validation is used, but in this case it is impossible to guarantee the independence of the obtained measurements, which does not allow the use of the Student's t-test. There are criteria that do not require independent measurements, but they have less power. For additive metrics, a technique is proposed in this paper, when a test sample is divided into N parts, on each of which the values of the metrics are calculated. Since the value on each part is obtained as the sum of independent random variables, according to the central limit theorem, the obtained metrics values on each of the N parts are realizations of the normally distributed random variable. To estimate the required sample size, it is proposed to use normality tests and build quantile– quantile plots. You can then use a modification of the Student's t-test to conduct a statistical test comparing the mean values of the metrics. A simplified approach is also considered, in which confidence intervals are built for the base model. A model whose metric values do not fall into this interval works differently from the base model. This approach reduces the amount of computations needed, however, an experimental analysis of the binary cross-entropy metric for CTR (Click-Through Rate) prediction models showed that it is more rough than the first one.
Subfields: Artificial Intelligence, Control and Systems Engineering, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
- Kyunghyun Cho
In Control and Systems Engineering
- G. Em Karniadakis
- Ameya Dilip Jagtap
- S. Murray
- Enrico Camporeale
- Hadi Meidani
- Mohammad Amin Nabian
- Paris Perdikaris
- Ulisses M. Braga-Neto
In Optimization & Training
- Elad Hoffer
- Jeremy Howard
- Suvrit Sra
- Daniel Soudry
- G. Em Karniadakis
- C. Freeman
- Sashank J. Reddi
- Matthias Hein
Citations in the corpus
- Universal Language Model Fine-tuning for Text Classification (2018, background): “For context, the regular stochastic gradient descent (SGD) update of a model’s parameters θ at time step t looks like the following (Ruder, 2016):”
- Fine-tuned Language Models for Text Classification (2018, background): “For context, the regular stochastic gradient descent (SGD) update of a model’s parameters θ at time step t looks like the following (Ruder, 2016): where η is the learning rate and ∇ θ J ( θ ) is the gradient with regard to the model’s objective function.”
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, background): “Yet, many current works still use simple variants of SGD (Ruder, 2016) for all or part of the optimization process (Wu et al., 2016).”
- On Layer Normalization in the Transformer Architecture (2020, methodology): “To study the first aspect, we train the model with the Adam optimizer (Kingma & Ba, 2014) and the vanilla SGD optimizer (Ruder, 2016) respectively.”
- Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, positive): “Since this is hard to handle with a global learning rate, it computes adaptive learning rates for each parameter [22].”
- Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD (2018, neutral): “For a detailed survey of different SGD variants, refer to [Ruder, 2016].”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “In most cases, Adam can act as the default optimizer (Ruder, 2016) because it is a more reliable improvement based on RMSprop.”
- Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, neutral): “RMSProp [41] altered the learning rate of each layer by dividing the square root of its exponential moving average.”
- Cyclical Learning Rates for Training Neural Networks (2015, background): “A summary and tutorial of adaptive learning rates can be found in a recent paper by Ruder [20].”
- Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator. (2017, methodology): “We can then employ stochastic gradient descent and its variants 32 , where the gradient ∇ θ J is replaced by randomly sampled unbiased estimators.”
- Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “Many other hand-designed optimizer architectures have been proposed (Ruder, 2016; Zeiler, 2012; Reddi et al., 2018; You et al., 2019; Liu et al., 2019), but their practical benefits are small in most situations (Schmidt et al.”
- Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks (2020, methodology): “The stochastic gradient descent (SGD) algorithm is widely used in machine learning community see [16] for a complete survey.”
- Recent Advances in Recurrent Neural Networks (2017, methodology): “A detailed comparison is available in [18].”
- Understanding and Co-designing the Data Ingestion Pipeline for Industry-Scale RecSys Training (2021, methodology): “We train recommendationmodels through a process called mini-batch stochastic gradient descent (SGD) [44], similar to most vision and natural language processing (NLP) models.”
- A high-bias, low-variance introduction to Machine Learning for physicists (2018, neutral): “The reader may also wish to consult useful reviews that cover these topics (Ruder, 2016) and this blog http://ruder.io/ optimizing-gradient-descent/.”