The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

By Bob Carpenter, Matthew D. Hoffman, Marcus A. Brubaker, Daniel Lee, Peter Li, Michael Betancourt. 2015

131 citations recorded by the source corpus.

As computational challenges in optimization and statistical inference grow ever harder, algorithms that utilize derivatives are becoming increasingly more important. The implementation of the derivatives that make these algorithms so powerful, however, is a substantial user burden and the practicality of these algorithms depends critically on tools like automatic differentiation that remove the implementation burden entirely. The Stan Math Library is a C++, reverse-mode automatic differentiation library designed to be usable, extensive and extensible, efficient, scalable, stable, portable, and redistributable in order to facilitate the construction and utilization of such algorithms. Usability is achieved through a simple direct interface and a cleanly abstracted functional interface. The extensive built-in library includes functions for matrix operations, linear algebra, differential equation solving, and most common probability functions. Extensibility derives from a straightforward object-oriented framework for expressions, allowing users to easily create custom functions. Efficiency is achieved through a combination of custom memory management, subexpression caching, traits-based metaprogramming, and expression templates. Partial derivatives for compound functions are evaluated lazily for improved scalability. Stability is achieved by taking care with arithmetic precision in algebraic expressions and providing stable, compound functions where possible. For portability, the library is standards-compliant C++ (03) and has been tested for all major compilers for Windows, Mac OS X, and Linux.

Subfields: Computational Theory and Mathematics, Management Science and Operations Research, Statistics and Probability, Probabilistic & Bayesian Methods, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Management Science and Operations Research

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Numerical Methods & Scientific Computing

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “The Stan library (Carpenter et al., 2015) implements gradient estimation through ODE solutions using forward sensitivity analysis.”
The Variational Gaussian Process (2015, methodology): “For gradients of the model log-likelihood, we employ convenient differentiation tools such as those in Stan and Theano (Carpenter et al., 2015; Bergstra et al., 2010).”
Automatic differentiation in machine learning: a survey (2015, methodology): “, 2016) implements automatic Bayesian inference based on HMC and the No-U-Turn sampler (NUTS) (Hoffman and Gelman, 2014) and uses reverse mode AD for the calculation of gradients for both HMC and NUTS (Carpenter et al., 2015).”
Automatic Differentiation Variational Inference (2016, methodology): “The added complexity of automatic differentiation over analytic gradients is roughly constant (Carpenter et al., 2015; Baydin et al., 2015).”
A review of automatic differentiation and its efficient implementation (2018, methodology): “Carpenter et al. (2015) corroborate the advantage of using expression templates by testing two implementations of a function that returns the log sum of exponents.”
Pathfinder: Parallel quasi-Newton variational inference (2021, unknown): “Although we summarize both log density and gradient costs for completeness, gradients typically consume closer to 80% of overall compute cost for Stan phase I and ADVI when calculated with automatic differentiation (Carpenter et al., 2015).”
Advances in Neural Rendering (2021, background): “A powerful AD library for C++ is STAN [CHB ∗ 15].”
TensorFlow Distributions (2017, background): “Stan Math [10] is a C++ templated library for numerical and statistical functions, and with automatic differentiation as the backend for the Stan probabilistic programming language [9].”