Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation

By M. Giles. 2008

128 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Statistical and Nonlinear Physics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f9e70230ac5a38bb2fe175def10d78fd8624f53c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

1. Gaetan K. W. Kenway
2. Charles A. Mader
3. Ping He
4. Joaquim R. Martins
5. Neil D. Lawrence
6. Jia Deng
7. Matthias W. Seeger
8. Bo Lang

In Statistical and Nonlinear Physics

1. Joaquim R. Martins
2. Gaetan K. W. Kenway
3. Charles A. Mader
4. Ping He
5. Neil D. Lawrence
6. Jia Deng
7. Matthias W. Seeger
8. Bo Lang

In Signal Processing

1. Zhenwen Dai
2. Joaquim R. Martins
3. Neil D. Lawrence
4. Jia Deng
5. Matthias W. Seeger
6. Bo Lang
7. Cristian Sminchisescu
8. Joel A. E. Andersson

Citations in the corpus

• Training Deep Networks with Structured Layers by Matrix Backpropagation (2015, methodology): “We call matrix backpropagation (MBP) the use of matrix calculus[49, 51, 52] to map between the partial derivatives ∂L ( l +1) ∂ x l and ∂L ( l ) ∂ x l − 1 at two consecutive structured layers.”
• Decorrelated Batch Normalization (2018, methodology): “We achieve this by using the fact that eigen-decomposition is differentiable, and its derivatives can be obtained using matrix differential calculus, as shown by prior work [15, 26].”
• A General-Purpose Software Framework for Dynamic Optimization (Een algemene softwareomgeving voor dynamische optimalisatie) (2013, background): “For details on how to derive this and other derivative propagation rules, we refer to [89,90].”
• A review of automatic differentiation and its efficient implementation (2018, methodology): “Finally, (Giles, 2008) summarizes several results to efficiently compute ma-20 derivatives, including matrix inverses, determinants, matrix exponentials, and Cholesky decompositions.”
• Effective adjoint approaches for computational fluid dynamics (2019, methodology): “In addition, to save memory, we switch on the checkpointing and use symbolic differentiation for the embedded linear solvers [182, 183].”
• Auto-Differentiating Linear Algebra (2017, methodology): “A number of backward expressions for linear algebra are given in [8], among them the one we use for the symmetric eigendecomposition here.”
• Matrix Backpropagation for Deep Networks with Structured Layers (2015, methodology): “We call matrix backpropagation (MBP) the use of matrix calculus[12, 32, 14] to map between the partial derivatives ∂L ∂xl and ∂L ∂xl−1 at two consecutive layers.”