Compiling Fast Partial Derivatives of Functions Given by Algorithms

By B. Speelpenning. 1980

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “1), a method for multilayer threshold NNs (Bobrowski, 1978), and a computer program for automatically deriving and implementing BP for given differentiable systems (Speelpenning, 1980).”
Automatic differentiation in machine learning: a survey (2015, background): “Speelpenning (1980) discovered reverse mode AD and gave the first implementation that was actually automatic, in the sense of accepting a specification of a computational process written in a general-purpose programming language and automatically performing the reverse mode transformation.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “To maximize differentiable objective functions of SL and UL, NN researchers almost invariably use backpropagation (BP) [125, 30, 52] in discrete graphs of nodes with differentiable activation functions [151, 265][245, Sec.”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, background): “The back-propagation algorithm (Rumelhart & Hinton, 1986), also known as reverse-mode automatic differentiation (Speelpenning, 1980; Rall, 1981) computes exact gradients of deterministic, differentiable objective functions.”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, methodology): “An image processing algorithm that can be defined as a Direct Acyclic Graph (DAG) structure can, thanks to the reverse-mode [53] auto-differentiation [19], compute gradients via backpropagation [27].”
Asymptotically exact inference in differentiable generative models (2016, methodology): “When applying reverse-mode AD [89, 54] to a function h : R → R the Jacobian Jh can be calculated at an operation-count cost which is at most cL times the corresponding cost of evaluating the function h itself.”
Nonstandard Interpretations of Probabilistic Programs for Efficient Inference (2011, background): “In reverse mode AD [25], computation of the derivative proceeds by propagating sensitivities of the output toward the input.”
Automatic differentiation in PyTorch (2017, background): “PyTorch, like most other deep learning libraries, supports reverse-mode [6] automatic differentiation [2] of scalar functions (or vector-Jacobian products of functions with multiple outputs), the most important form of automatic differentiation for deep learning applications which usually…”