Taylor expansion of the accumulated rounding error

By S. Linnainmaa. 1976

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “…backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master’s thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (Griewank, 2012), where…”
The hardware lottery (2020, background): “Most of the algorithmic components to make deep neural networks work had already been in place for a few decades: backpropagation (invented in 1963 (K & Piske, 1963), reinvented in 1976 (Linnainmaa, 1976), and again in 1988 (Rumelhart et al.”
Asymptotically exact inference in differentiable generative models (2016, methodology): “When applying reverse-mode AD [89, 54] to a function h : R → R the Jacobian Jh can be calculated at an operation-count cost which is at most cL times the corresponding cost of evaluating the function h itself.”
My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013 (2013, methodology): “However, efﬁcient error backpropagation (BP) in arbitrary, possibly sparse, NN-like networks apparently was ﬁrst described by Linnainmaa in 1970 [59, 60] (he did not refer to NN though).”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, methodology): “Backpropagation dates back to the works of Dreyfus (1962), Linnainmaa (1976) and Rumelhart, Hinton, and Williams (1986), while its extension to RNNs termed Backpropagation through time (BPTT) was presented in Werbos (1988, 1990).”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, background): “tworks start in the 1940s and gained increasing attention and developments throughout the 1970s and 1980s. Some of the most important concepts, such as automatic differentiation [14], backpropagation [15] [16], max-pooling [17] are proposed during this time. During the 1990s and 2000s, researches in”
An Introductory Review of Deep Learning for Prediction Models With Big Data (2020, methodology): “However, the method itself is older (see e.g., Linnainmaa, 1976).”
Experimentally realized in situ backpropagation for deep learning in photonic neural networks (2022, methodology): “These improvements on an already versatile photonic hardware architecture enable backpropagation-based machine learning implemented entirely using optical measurement to optimize programmable phase shifters in PNNs.”
Machine Learning in Agriculture: A Review (2018, methodology): “The learning algorithms commonly used in ANNs include the radial basis function networks [47], perceptron algorithms [48], back-propagation [49], and resilient back-propagation [50].”
Moment Tensor Potentials: A Class of Systematically Improvable Interatomic Potentials (2015, methodology): “The forces are computed by reverse-mode differentiation of the energy with respect to the atomic positions [22].”
Recurrent Highway Networks (2016, background): “Unfortunately, increased depth represents a challenge when neural network parameters are optimized by means of error backpropagation [7–9].”