BackPACK: Packing more into backprop
By Felix Dangel, Frederik Kunstner, Philipp Hennig. 2019
117 citations recorded by the source corpus.
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient. Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in theory, be computed efficiently, and at the same time as the gradient. While these quantities are of great interest to researchers and practitioners, current deep learning software does not support their automatic calculation. Manually implementing them is burdensome, inefficient if done naively, and the resulting code is rarely shared. This hampers progress in deep learning, and unnecessarily narrows research to focus on gradient descent and its variants; it also complicates replication studies and comparisons between newly developed methods that require those quantities, to the point of impossibility. To address this problem, we introduce BackPACK, an efficient framework built on top of PyTorch, that extends the backpropagation algorithm to extract additional information from first-and second-order derivatives. Its capabilities are illustrated by benchmark reports for computing additional quantities on deep neural networks, and an example application by testing several recent curvature approximations for optimization.
Subfields: Artificial Intelligence, Computational Mathematics, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Agustinus Kristiadi
- Erik A. Daxberger
- Philipp Hennig
- Matthias Bauer
- Runa Eschenhagen
- Alexander Immer
- Matthias Hein
- Francesco Croce
In Computational Mathematics
In Computational Mechanics
- Agustinus Kristiadi
- Matthias Hein
- Runa Eschenhagen
- Matthias Bauer
- Philipp Hennig
- Alexander Immer
- Erik A. Daxberger
Citations in the corpus
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “That is, we find the Kronecker factorization of the Hessian H−1 ≈ V−1 ⊗U−1 via automatic differentiation (Dangel et al., 2020).8 Then by definition of a matrix Gaussian (Gupta & Nagar, 1999), we immediately obtain the posteriorMN (W|WMAP,U,V).”
- Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “As we discuss below, the Hessian computation can be offloaded to recently advanced automatic differentiation libraries [21].”