Auto-Differentiating Linear Algebra
By Matthias W. Seeger, Asmus Hetzel, Zhenwen Dai, Neil D. Lawrence. 2017
43 citations recorded by the source corpus.
Development systems for deep learning, such as Theano, Torch, TensorFlow, or MXNet, are easy-to-use tools for creating complex neural network models. Since gradient computations are automatically baked in, and execution is mapped to high performance hardware, these models can be trained end-to-end on large amounts of data. However, it is currently not easy to implement many basic machine learning primitives in these systems (such as Gaussian processes, least squares estimation, principal components analysis, Kalman smoothing), mainly because they lack efficient support of linear algebra primitives as differentiable operators. We detail how a number of matrix decompositions (Cholesky, LQ, symmetric eigen) can be implemented as differentiable operators. We have implemented these primitives in MXNet, running on CPU and GPU in single and double precision. We sketch use cases of these new operators, learning Gaussian process and Bayesian linear regression models. Our implementation is based on BLAS/LAPACK APIs, for which highly tuned implementations are available on all major CPUs and GPUs.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jan Gasthaus
- Bernie Wang
- Tim Januschowski
- Matthias W. Seeger
- Cédric Archambeau
- Danielle C. Maddix
- Alex Smola
- Rodolphe Jenatton
Citations in the corpus
- High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “ining or NaNs appear during training because of unstable models. Finally, RNNs are combined with Zone-out regularization [9] and residual connections and MXNet is used as the neural network framework [15]. L Open-source implementation of our model We re-implemented the model described in this paper ”
- Deep Factors for Forecasting (2019, methodology): “Our algorithms are implemented in MXNet Gluon (Chen et al., 2015), and make extensive use of its Linear Algebra library (Seeger et al., 2017; Zhenwen et al., 2018).”
- Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start (2017, methodology): “In particular, we leverage the backward operator for the Cholesky decomposition [19].”