Additive Gaussian Processes

By David Kristjanson Duvenaud, Hannes Nickisch, Carl E. Rasmussen. Neural Information Processing Systems, 2011

365 citations recorded by the source corpus.

We introduce a Gaussian process model of functions which are additive. An additive function is one which decomposes into a sum of low-dimensional functions, each depending on only a subset of the input variables. Additive GPs generalize both Generalized Additive Models, and the standard GP models which use squared-exponential kernels. Hyperparameter learning in this model can be seen as Bayesian Hierarchical Kernel Learning (HKL). We introduce an expressive but tractable parameterization of the kernel function, which allows efficient evaluation of all input interaction terms, whose number is exponential in the input dimension. The additional structure discoverable by this model results in increased interpretability, as well as state-of-the-art predictive power in regression tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5f9ce48983b571aa69706cffc7ace33f352ac59a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Max-value Entropy Search for Efficient Bayesian Optimization (2017, methodology): “Additive Gaussian processes (add-GP) were proposed in (Duvenaud et al., 2011), and analyzed in the BO setting in (Kandasamy et al., 2015).”
Bayesian Optimization with Tree-structured Dependencies (2017, background): “Several authors attempted to design covariance functions that are aware of the structure: Duvenaud et al. (2011) consider kernels with an additive structure, while Swersky et al. (2014a); Hutter & Osborne (2013) introduce the Arc-kernel.”
Structure Discovery in Nonparametric Regression through Compositional Kernel Search (2013, methodology): “To evaluate the predictive accuracy of our method in a high-dimensional setting, we extended the comparison of (Duvenaud et al., 2011) to include our method.”
Differentiable Compositional Kernel Learning for Gaussian Processes (2018, methodology): “Even for experts, choosing GP kernel structures remains a dark art. can be done by imposing a speciﬁc kind of structure: e.g., Bach (2009); Duvenaud et al. (2011) learned kernel structures which were additive over subsets of the variables.”
Convolutional Gaussian Processes (2017, methodology): “We use the marginal likelihood to automatically weigh how much of the process should be explained by each of the components, in the same way as is done in other additive models [27, 35].”
Deep convolutional Gaussian processes (2018, background): “Deep Gaussian processes boost performance by modelling networks of GP nodes (Duvenaud et al., 2011; Sun et al., 2018) or by mapping inputs through multiple Gaussian process ’layers’ (Damianou and Lawrence, 2013; Salimbeni and Deisenroth, 2017).”
High Dimensional Bayesian Optimisation and Bandits via Additive Models (2015, background): “Duvenaud et al. (2011) assume a sum of functions of all combinations of lower dimensional coordinates.”
Additive Approximations in High Dimensional Nonparametric Regression via the SALSA (2016, methodology): “…( BF ) (Hastie & Tibshirani, 1990), Multivariate Adaptive Regression Splines ( MARS ) (Friedman, 1991), Component Selection and Smoothing ( COSSO ) (Lin & Zhang, 2006), Sparse Additive Models ( SpAM ) (Raviku-mar et al., 2009) and Additive Gaussian Processes ( Add-GP ) (Duvenaud et al., 2011).”
Avoiding pathologies in very deep networks (2014, background): “This model class was studied by Duvenaud et al. (2011), who showed that exact inference in these models can be performed in O(N(2)D(2) +N(3)).”
Learning Stationary Time Series using Gaussian Processes with Nonparametric Kernels (2015, background): “…[4] [5], but although kernel design is widely acknowledged as pivotal, it typically proceeds via a " black art " in which a particular functional form is hand-crafted using intuitions about the application domain to build a kernel using simpler primitive kernels as building blocks (e.g. [6]).”
Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics (2016, neutral): “2013) or encoding additional structure in the kernel (Duvenaud et al. 2011).”
ANOVA kernels and RKHS of zero mean functions for model-based sensitivity analysis (2011, neutral): “13, ANOVA kernels can also be seen as a sum of separable kernels with increasing interaction orders [6].”
Recent Advances in Bayesian Optimization (2022, background): “The addictive structure has been used in addictive GPs [39].”
Polynomial-Chaos-based Kriging (2015, neutral): “The latest developments in Kriging are contributed in the aspects of optimal estimation of the hyper-parameters [67, 75, 76], the use of adaptive kernels [77, 78], and the use of additive auto-correlation kernels [79–81].”