Automatic differentiation in machine learning: a survey

By Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Radul, Jeffrey Mark Siskind. 2015

3,658 citations recorded by the source corpus.

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply “auto-diff”, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other’s results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names “dynamic computational graphs” and “differentiable programming”. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main imple- mentation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms “autodiff”, “automatic differentiation”, and “symbolic differentiation” as these are encountered more and more in machine learning settings.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Neural Ordinary Differential Equations (2018, positive): “While use of vector-Jacobian products for solving the adjoint method has been explored in optimal control (Andersson, 2013), we explore a fully general integration of black-box ODE solvers into automatic differentiation (Baydin et al., 2018) and highlight its potential in deep learning and generativ”
An Introduction to Probabilistic Programming (2018, methodology): “In order to compute the gradients of the potential function, we will use reverse-mode automatic differentiation (AD) (Griewank and Walther, 2008; Baydin et al., 2015), which is the technique that forms the basis for modern deep learning systems such as TensorFlow (Abadi et al.”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, methodology): “Distributed Learning has become a wide-spread framework for large-scale model training (Dean et al., 2012; Li et al., 2014a;b; Baydin et al., 2017; Zhang et al., 2017; Agarwal et al., 2010; Recht et al., 2011), in which a server is leveraging the compute power of many devices by aggregating local…”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, background): “Then the following facts hold true (Griewank & Walther, 2008) (see also Baydin et al. (2015) for a shorter account): (i) For any vector r ∈ R n , the product J F r can be evaluated in time O ( c ( n, p )) and requires space O ( s ( n, p )) using forward-mode AD.”
Physics informed deep learning for computational elastodynamics without labeled data (2020, background): “The reason why DNN is well qualified as the solution approximator includes: (1) its extraordinary approximation capability proven by the universal approximation theorem [49], (2) its infinite continuity property [14] with proper activation function, (3) the derivatives that appears in the PDEs can b”
Understanding and mitigating gradient pathologies in physics-informed neural networks (2020, methodology): “Following the original work of [12], we then proceed by approximating u(x, t) by a deep neural network fθ(x, t), and define the residual of equation 2 as rθ(x, t) := ∂ ∂t fθ(x, t) +Nx[fθ(x, t)], (3) where the partial derivatives of the neural network representation with respect to the space and time”
Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations (2018, methodology): “Confusion can arise because automatic differentiation does in fact provide numerical values of derivatives (as opposed to derivative expressions) and it does so by using symbolic rules of differentiation (but keeping track of derivative values as opposed to the resulting expressions), giving it a tw”
Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations (2017, methodology): “This network can be derived by applying the chain rule for differentiating compositions of functions using automatic differentiation [11].”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “For further information on automatic differentiation see the work of Neidinger (2010, Section 7) and Baydin et al. (2015). For more in depth discussion of the backpropagation algorithm and computation graphs (also called flow graphs) see the work of Bengio et al. (2015, Section 6.4), LeCun et al. (1”
Adversarial Uncertainty Quantification in Physics-Informed Neural Networks (2018, neutral): “recent progress in automatic differentiation [31, 32].”
Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations (2017, methodology): “We exploit recent developments in automatic differentiation [13] – one of the most useful but perhaps underused techniques in scientific computing – to differentiate neural networks with respect to their input coordinates and model parameters to obtain physics informed neural networks.”
Online Learning Rate Adaptation with Hypergradient Descent (2017, background): “Our initial conceptualization of the learning rate adaptation problem was one of automatic differentiation (Baydin et al., 2015).”
The Generalized Reparameterization Gradient (2016, methodology): “The gradient ∇zf(z) depends on the model, but it can be computed using automatic differentiation tools (Baydin et al., 2015).”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, positive): “…that it makes it possible to compute efﬁciently the gradient of f T , which we call hypergradient , either in time or in memory (Maclaurin et al., 2015; Franceschi et al., 2017), by making use of reverse or forward mode algorithmic differentiation (Griewank and Walther, 2008; Baydin et al., 2017).”
Sparse Variational Inference: Bayesian Coresets from Scratch (2019, background): “But developments in automated tools—e.g., automatic differentiation [5, 6], “black-box” gradient estimates [7], and Hamiltonian transition kernels [8, 9]—have obviated much of this expert input, greatly expanding the repertoire of Bayesian models accessible to practitioners.”