Increased rates of convergence through learning rate adaptation

By Robert A. Jacobs. 1987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “…BP step size adapt a global learning rate (Battiti, 1989; Lapedes & Farber, 1986; LeCun, Simard, & Pearlmutter, 1993; Vogl, Mangis, Rigler, Zink, & Alkon, 1988; Yu, Chen, & Cheng, 1995), while others compute individual learning rates for each weight (Jacobs, 1988; Silva & Almeida, 1990).”
Automated optimized parameters for T-distributed stochastic neighbor embedding improve visualization and analysis of large datasets (2019, neutral): “The step size in t-SNE gradient descent is updated at each iteration per Jacobs adaptive learning rate scheme 22 .”
openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding (2019, methodology): “Optimization is performed using batch gradient descent using the delta-bar-delta update rule (Jacobs 1988).”
On the momentum term in gradient descent learning algorithms (1999, background): “This approach has been found to speed up training (Jacobs, 1988).”
Barnes-Hut-SNE (2013, methodology): “We run a gradient-descent optimizer for 1 , 000 iterations, setting the initial step size to 200 ; the step size is updated during the optimization use the scheme of [13].”
Online Learning Rate Adaptation with Hypergradient Descent (2017, methodology): “roducts [17]. Other work that merits mention include RPROP [20], where local adaptation of weight updates are performed by using only the temporal behavior of the gradient’s sign, and Delta-Bar-Delta [12], where the learning rate is varied based on a sign comparison between the current gradient and ”
Evolving artificial neural networks (1999, background): “This evolutionary approach is different from the nonevolutionary one such as offered by Jacobs [273] because the simultaneous evolution of both algorithmic parameters and architectures facilitates exploration of interactions between the learning algorithm and architectures such that a near-optimal…”
From biological models to the evolution of robot control systems (2003, background): “The idea of variable neural plasticity is also quite common in the ̄eld of arti ̄cial neural networks, where modellers have found it bene ̄cial to vary their network-learning rates during the course of training (Jacobs 1988).”
Gradient calculations for dynamic recurrent neural networks: a survey (1995, methodology): “One technique that has proven useful in this particular situation is that of [134] which was applied by Fang and Sejnowski to the single gure eight problem perturbed in gure 9 with great success by [135].”
Partial BFGS Update and Efficient Step-Length Calculation for Three-Layer Neural Networks (1997, methodology): “1993), learning rate adaptation rules (Jacobs 1988; Silva and Almeida 1990), smart algorithms such as QuickProp (Fahlman 1988) and RPROP (Riedmiller and Braun 1993), and second-order learning algorithms (Watrous 1987; Barnard 1992; Battiti 1992; Møller 1993b) based on nonlinear optimization techniqu”
Local Gain Adaptation in Stochastic Gradient Descent (1999, methodology): “Unfortunately most of the existing gain adaptation algorithms for neural networks adapt only a single, global learning rate [6, 7], can be used only in batch training [8, 9, 10, 11, 12, 13], or both [14, 15, 16, 17].”
Accelerated Backpropagation Learning: Two Optimization Methods (1989, background): “…term (and a parameter a), leading to the following rule: (1.4) (1.5) Recent ly, an overview of heuristics employed to accelerate backpropagation has been presented in [10], where it is suggest ed t hat each weight should be given its own lea rning rate, changing over time during the comp utation.”
Neural Networks for Real-Time Traffic Signal Control (2006, methodology): “Given the favorable properties of SPSA mentioned previously , it has been used to update the weights of an NN [6], [18], [19] in order to find the minimum point of the loss function.”