Structure Discovery in Nonparametric Regression through Compositional Kernel Search

By David Kristjanson Duvenaud, James Robert Lloyd, Roger Baker Grosse, Josh Tenenbaum, Zoubin Ghahramani. 2013

559 citations recorded by the source corpus.

Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirrors the scientific discovery process. The learned structures can often decompose functions into interpretable components and enable long-range extrapolation on time-series datasets. Our structure search method outperforms many widely used kernels and kernel combination methods on a variety of prediction tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Bayesian Learning of Kernel Embeddings (2016, background): “This is noteworthy for its success and general applicability even for learning complicated combinations of kernels (Duvenaud et al., 2013) or rich kernel families (Wilson and Adams, 2013).”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “People have built more expressive functions by automatically combining kernels together [14] as well as by composing GPs together [13, 11, 15].”
Hidden physics models: Machine learning of nonlinear partial differential equations (2017, background): “For a more systematic treatment of the kernelselection problem we would like to refer the readers to [17, 18, 19].”
The future of employment: How susceptible are jobs to computerisation? (2017, background): “Duvenaud et al. (2013) provide an example of automating the core creative task required in order to perform statistics, that of designing models for data.”
Automatic Construction and Natural-Language Description of Nonparametric Regression Models (2014, methodology): “Our procedure is based on the model construction method of Duvenaud et al. (2013) which automatically decomposed models but components were interpreted manually and the space of models searched over was smaller than that in this work.”
Venture: a higher-order probabilistic programming platform with programmable inference (2014, background): “For example, a prior where resimulation recovers some of the search moves from (Duvenaud et al., 2013) may be expressible by separately generating symbolic expressions and the contents of the environment into which they are evaluated.”
Avoiding pathologies in very deep networks (2014, background): “Such feature maps can capture rich structure (Duvenaud et al., 2013), and can enable many types of generalization, such as translation and rotation invariance in images (Kondor, 2008).”
Machine learning of linear differential equations using Gaussian processes (2017, methodology): “Although in this study the model form was assumed to be known, another potential extension could pursue learning the model form itself using ideas from compositional kernel search [15].”
Identifying Causal Structure in Large-Scale Kinetic Systems (2018, methodology): “ in Aijo and Lahdesmaki [2009], Guo et al. [2014], Casadiego et al. [2017] and also methods based on general time-series models, which focus on non-parametric kernel methods [Grosse et al., 2012, Duvenaud et al., 2013]. For a review of network inference methods with a focus on systems biology, we re”
Gaussian process regression for forecasting battery state of health (2017, background): “[19] provides a more detailed discussion of kernel composition and also addresses the issue of automating the choice of kernels.”
Variational Auto-encoded Deep Gaussian Processes (2015, background): “…and combinations of covariance functions can lead to powerful models (Durrande et al., 2011; G ¨ onen & Alpaydin, 2011; Hensman et al., 2013; Duvenaud et al., 2013; Wilson & Adams, 2013), the assumption about joint normal distribution of instantiations of the latent function remains;…”
A temporal model of text periodicities using Gaussian Processes (2013, methodology): “The ﬂexibility of the GP framework allows us to combine kernels (e.g. SE · P S or P S + Lin ) in order to identify a combination of trends (Duvenaud et al., 2013; G¨onen and Alpaydin, 2011).”
Using Regression Kernels to Forecast A Failure to Appear in Court (2014, background): “Still, this coupling of kernel to application is usually justified by little more than hunch or craft lore (Gross et al., 2012; Duvenaud et al., 2013).”
From Dependence to Causation (2016, background): “Duvenaud et al. (2013) proposes a genetic algorithm that explores these kind of compositions to evolve complex kernels with interpretable meanings.”
Gaussian Processes for Independence Tests with Non-iid Data in Causal Inference (2015, background): “Recent work suggests that complicated time series dynamics can be automatically fit through combinations of covariances [Wilson and Adams 2013; Duvenaud et al. 2013].”
A tutorial on Gaussian process regression: Modelling, exploring, and exploiting functions (2016, methodology): “General solutions to this are to constructmore complicated kernels from a set of relatively simple base kernels (as shown above) and to search the kernel space by proposing and checking new kernel combinations (Duvenaud et al., 2013), or to define the kernel in a non-parametric manner by using a non”