Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition

By Andreas Griewank, Andrea Walther. 2000

3,365 citations recorded by the source corpus.

Subfields: Education, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “This can be seen as a form of selective gradient checkpointing [10, 36].”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, methodology): “7 Computational Complexity Assuming Algorithmic Differentiation [Griewank and Walther, 2008] Time complexity of Φt : CT = g(d ,m) Space complexity of Φt : CS = (d ,m).”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “In this paper, we use implicit differentiation (Dontchev & Rockafellar, 2009; Griewank & Walther, 2008) and techniques from matrix differential calculus (Magnus & Neudecker, 1988) to derive the gradients from the KKT matrix of the problem.”
The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo (2011, background): “Computing the gradient for a complex model is at best tedious and at worst impossible, but this requirement can be made less onerous by using automatic diﬀerentiation (Griewank and Walther, 2008).”
Automatic differentiation in machine learning: a survey (2015, methodology): “In terms of computational complexity, AD ensures that the amount of arithmetic goes up by no more than a small constant (Griewank and Walther, 2008).”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, methodology): “An image processing algorithm that can be defined as a Direct Acyclic Graph (DAG) structure can, thanks to the reverse-mode [53] auto-differentiation [19], compute gradients via backpropagation [27].”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, positive): “…that it makes it possible to compute efﬁciently the gradient of f T , which we call hypergradient , either in time or in memory (Maclaurin et al., 2015; Franceschi et al., 2017), by making use of reverse or forward mode algorithmic differentiation (Griewank and Walther, 2008; Baydin et al., 2017).”
Topology Optimization of Turbulent Flows (2018, methodology): “To this end, automatic differentiation (AD) [21, 22] will likewise be employed for calculating the exact gradients of the optimization objective and constraints in the topology optimization problems considered.”
Fine-Tuning Language Models with Just Forward Passes (2023, methodology): “See Appendix B. Forward Auto-Differentiation Note that z ⊤ ∇L ( θ ; B ) is a Jacobian-vector product (JVP), which can be computed in parallel with an inference pass with excess memory consumption equivalent to that of the largest activation in the network [40].”
Recent advances in surrogate-based optimization (2009, background): “Most likely to be of use is gradient information found via algorithmic differentiation (AD) [29] (also known as automatic differentiation, though the term ‘automatic’ can instill false hope, since manual intervention is required in most cases), which requires access to the function source code, and ”
A General-Purpose Software Framework for Dynamic Optimization (Een algemene softwareomgeving voor dynamische optimalisatie) (2013, methodology): “The major approaches for this are the sparse-vector approach [150] and the related index domain approach [101], where sets of indices are propagated through the algorithm, as well as the bitvector approach, where the AD algorithms – forward or reverse – are evaluated using a boolean data type.”
Lagrangian data assimilation for river hydraulics simulations (2006, background): “A large part of this extensive task can be automated using algorithmic differentiation [6].”
Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces (2013, positive): “Given continued interest in gradient computations based on adjoint methods [5, 22] and algorithmic differentiation [20], it is not unreasonable to assume that one has access to the gradient of the function.”