Efficient BackProp

By Yann LeCun, L. Eon Bottou, Genevieve B. Orr, K. Müller. 2012

3,522 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Optimization & Training

In Signal Processing

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].”
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “As shown in (LeCun et al., 1998b), such normalization speeds up convergence, even when the features are not decorrelated.”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “In [20, 25, 35] the analysis of FIM was a motivation of centering the nonlinear responses of each layer, which eases training of symmetric, sigmoid-like units.”
Fully convolutional networks for semantic segmentation (2014, neutral): “By contrast, prior work randomly samples patches over a full dataset [27, 2, 7, 28, 9], potentially resulting in higher variance batches that may accelerate convergence [22].”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “These successes have been achieved despite the noted difficulty of training such deep architectures [7, 8, 9, 10, 11].”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, background): “In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transformations that speed up learning [8, 11, 12].”
No more pesky learning rates (2012, methodology): “While the practical advantages of SGD for machine learning applications have been known for a long time (LeCun et al., 1998), interest in SGD has increased in recent years due to the ever-increasing amounts of streaming data, to theoretical optimality results for generalization error (Bottou &…”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “Le Cun et al. (1998) suggest that a batch size of one can result in better solutions because the noisier updates allow for the possibility of escaping from local minima in a descent algorithm.”
Density estimation using Real NVP (2016, methodology): “This class of algorithms tractably models the joint distribution by decomposing it into a product of conditionals using the probability chain rule according to a fixed ordering over dimensions, simplifying log-likelihood evaluation and sampling.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Therefore tanh has been preferred over logistic functions (LeCun et al., 1991; 1998).”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, methodology): “To improve convergence performance, we used the tanh output activation function (LeCun et al., 1998).”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “In practice, minimizing a large neural network’s training loss using SGD often results in models that generalize well to new data [3, 17, 27], making SGD an invaluable tool.”
Natural Language Processing (Almost) from Scratch (2011, methodology): “The local minima found by the training algorithm are usually good local minima, thanks to the oversized parameter space and to the noise induced by the stochastic gradient procedure (LeCun et al., 1998).”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “We initialize the underlying weights for these layers using LeCun initialization (LeCun et al., 2012).”
Deep Reinforcement Learning that Matters (2017, methodology): “By intuition, in gradient based methods (as used in most deep RL) a large and sparse output scale can result in problems regarding saturation and inefficiency in learning (LeCun et al. 2012; Glorot and Bengio 2010; Vincent, de Brébisson, and Bouthillier 2015).”
Representation Learning: A Review and New Perspectives (2012, background): “In some cases, such as in the convolutional neural network (LeCun et al., 1998b), we build this abstraction in explicitly via a pooling mechanism (see section 11.2).”
Data-dependent Initializations of Convolutional Neural Networks (2015, neutral): “Early approaches to data-driven initializations showed that whitening the activations at all layers can mitigate the vanishing gradient problem (LeCun et al., 1998), but it does not ensure all layers train at an equal rate.”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “This increases the amount of computation per iteration, which can be effectively distributed.”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, background): “) is the element-wise scaled hyperbolic tangent function (LeCun et al. (2012)): This function forces the gradients into the most non-linear value range and leads to a faster the training process than the basic hyperbolic tangent function.”