Forward-Mode Automatic Differentiation in Julia
By Jarrett Revels, Miles Lubin, Theodore Papamarkou. arXiv.org, 2016
445 citations recorded by the source corpus.
We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics.
Subfields: Computer Networks and Communications, Hardware and Architecture
View Paper: https://www.semanticscholar.org/paper/7bfe9b201b0f4e383d82431cbcb18ec9f300614c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Hardware and Architecture
Citations in the corpus
- Universal Differential Equations for Scientific Machine Learning (2020, methodology): “Through the integration with automatic differentiation, discrete adjoint sensitivity analysis [49, 50] is implemented through both tape-based reverse-mode [51] and source-to-source translation [52], with computational trade-offs between the two approaches.”
- Competitive Gradient Descent (2019, unknown): “jl, (Revels et al., 2016)) together with ReverseDiff(https://github.”
- Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism (2020, unknown): “However, PINNs promise to take this approach to a different level by using deep neural networks, which is made possible by the vast advances in computational capabilities and training algorithms since that time [7, 8], as well as the availability of automatic differentiation methods [9, 10].”