Backpropagation: past and future

By Paul J. Werbos. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “pled Newton methods and consider problems of a moderate dimension where it is computationally efficient to perform subsampled Hessian-vector operations (Werbos, 1988; Pearlmutter, 1994).”
Inversion of multilayer nets (1989, background): “Choosing A = 0 would cause a normal decay like the standard weight decay (see also Werbos 88 [ZO]).”
Model Selection: Beyond the Bayesian/Frequentist Divide (2010, methodology): “The same regularizer w 2 is used in ridge regression (Hoerl, 1962), " weight decay " neural networks (Werbos, 1988), regularized radialbasis function networks (Poggio and Girosi, 1990), Gaussian processes (MacKay, 1992), together with the square loss function.”
Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches (1992, methodology): “Many authors have discussed alternatives to equation 14 which can speed up convergence [21,24,25]”
Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities (2006, background): “(See also [ 32 ].) The easiest general approach is simply to note that the calculation of E(t) is itself an ordered system, even though some of its intermediate calculations are motivated by derivative calculation.”
Understanding Short-Horizon Bias in Stochastic Meta-Optimization (2018, methodology): “(9) While the Hessian is infeasible to construct explicitly, the Hessian-vector product in Equation 9 can be computed efficiently using reverse-on-reverse (Werbos, 1988) or forward-on-reverse automatic differentiation (Pearlmutter, 1994), in time linear in the cost of the forward pass.”
Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent (2002, methodology): “This fast Hessian-vector product (Pearlmutter, 1994; Werbos, 1988; Møller, 1993) can be used in conjunction with equation 1.”
A neural network approach for credit risk evaluation (2008, methodology): “There are algorithms specifically designed for a particular kind of neural networks, such as the backpropagation algorithm (Werbos, 1988) or general purpose algorithms, such as genetic algorithms (Mitchell, 1998) and simulated annealing (Kirkpatrick, Gelatt, & Vecchi, 1983).”