On optimization methods for deep learning

By Quoc V. Le, Jiquan Ngiam, Adam Coates, Ahbik Lahiri, Bobby Prochnow, Andrew Y. Ng. 2011

1,051 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, background): “(Le et al., 2011) compares the performance of stochastic gradient and quasi-Newton methods on neural network training, and ﬁnds both to be competitive.”
Representation Learning: A Review and New Perspectives (2012, methodology): “9). Finally, the debate still rages between using online methods such as stochastic gradient descent and using second-order methods on large minibatches (of several thousand examples) (Martens, 2010; Le et al., 2011a), with a variant of stochastic gradient descent recently winning an optimization ch”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, neutral): “Other applicable algorithms which are not based on SGD are L-BFGS and conjugate gradient [59].”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, neutral): “Performance of several competitive algorithms for training deep neural networks has been compared in [70].”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, background): “Continual learning remains a long-standing challenge for DNNs since these networks are typically trained with stationary training batches by stochastic gradient descent methods [19], which generally leads to an abrupt performance decrease on previously learned tasks as new tasks are learned.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “arger number of hidden units. Several approaches have been proposed to induce a sparse representation (or with more hidden units whose activation is closer to 0). One approach (Ranzato et al., 2008b; Le et al., 2011; Zou et al., 2011) is simply to penalize the L1 norm of the representationoranother ”
DeepMPC: Learning Deep Latent Features for Model Predictive Control (2015, methodology): “Learning System: We used the L-BFGS algorithm, shown to give strong results for deep learning methods [23], to optimize our model during learning.”
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond (2016, background): “Ngiam et al. (2011) has an in depth review of some of the proposals for approximating the Hessian of the loss function.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “When performing SGD, it is common to decrease the number of weight updates by computing the sample loss in minibatches (Algorithm 2), averaging the gradient with respect to subsets of the data [147].”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “Performance of several competitive algorithms for training deep neural networks has been compared in [130].”
Deep Learning for Health Informatics (2017, background): “Although more advanced variants of backpropagation [22] can solve this problem, they still result in a very slow learning process.”
Computational Phenotype Discovery Using Unsupervised Feature Learning over Noisy, Sparse, and Irregular Clinical Data (2013, methodology): “The weights and biases are typically learned with a convex optimization algorithm such as L-BFGS [51,52].”
Image Denoising and Inpainting with Deep Neural Networks (2012, methodology): “In both of the pre-training and fine-tuning stages, the loss functions are optimized with L-BFGS algorithm (a Quasi-Newton method) which, according to [22], can achieve fastest convergence in our settings.”