Automatic Construction and Natural-Language Description of Nonparametric Regression Models

By James Robert Lloyd, David Kristjanson Duvenaud, Roger Baker Grosse, Josh Tenenbaum, Zoubin Ghahramani. 2014

256 citations recorded by the source corpus.

This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural-language text. Our approach treats unknown regression functions nonparametrically using Gaussian processes, which has two important consequences. First, Gaussian processes can model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional structure of our language of models this allows us to automatically describe functions in simple terms. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state-of-the-art extrapolation performance evaluated over 13 real time series data sets from various domains.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Statistical Learning Theory

Citations in the corpus

Interpretable Distribution Features with Maximum Testing Power (2016, methodology): “Lloyd and Ghahramani (2015) used this witness function to compare the model output of the Automated Statistician (Lloyd et al., 2014) with a reference sample, yielding a visual indication of where the model fails.”
Deep Kernel Learning (2015, background): “Recent approaches (e.g., Wilson, 2014; Wilson and Adams, 2013; Lloyd et al., 2014; Yang et al., 2015) have demonstrated that one can develop more expressive kernel functions, which are indeed able to discover rich structure in data without human intervention.”
Stochastic Variational Deep Kernel Learning (2016, background): “Indeed many recent approaches have advocated building expressive kernel functions [e.g., 23, 10, 27, 26, 18, 32], and emerging research in this direction takes inspiration from deep learning models [e.g., 29, 6, 3].”
Automated Machine Learning: Methods, Systems, Challenges (2019, unknown): “Assuming further that all probability distributions of source nodes in the causal graph are non-Gaussian, [28] show that the causal structure is fully identifiable (all edges can be oriented).”
iBCM: Interactive Bayesian Case Model Empowering Humans via Intuitive Interaction (2015, background): “While explanations using highly sophisticated language of statistics may offer in-depth understanding for expert users [24], iBCM provides intuitive and consistent explanations for non-expert users.”
Interactive and interpretable machine learning models for human machine collaboration (2015, background): “While explanations using highly sophisticated language of statistics may offer in-depth understanding for expert users [93], iBCM provides intuitive explanations for non-expert users.”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “There are a growing number of packaged toolkits (e.g., WEKA (Hall et al. 2009) and Scikit-learn (Pedregosa et al. 2011)) and automated functionality (e.g., The Automatic Statistician (Lloyd et al. 2014)).”
ML-Plan: Automated machine learning via hierarchical planning (2018, methodology): “Ideally, the induction of models from data, including the data preprocessing, the choice of a model class, the training and evaluation of a predictor, the representation and interpretation of results, etc., would be automated to a large extent (Lloyd et al. 2014).”
Deep feature synthesis: Towards automating data science endeavors (2015, methodology): “The Automatic Statistician project automatically models regression problems and produces reports readable by humans [17].”
Functional Variational Bayesian Neural Networks (2019, background): “This includes stochastic processes with explicit densities, such as GPs which can model various structures like smoothness and periodicity (Lloyd et al., 2014; Sun et al., 2018).”
One button machine for automating feature engineering in relational databases (2017, unknown): “Automatic Statistician [9] is similar to the works just described but focuses more on time-series data and interpretation of the models in natural language.”
A tutorial on Gaussian process regression: Modelling, exploring, and exploiting functions (2016, background): “However, as different kernels encode different assumptions about the underlying functions, assessing which kernel describes the underlying function best can be used as a basis to interpret the modelled function (Lloyd et al., 2014).”
Statistical Model Criticism using Kernel Two Sample Tests (2015, methodology): “While it is clear that simple methods will fail to capture all of the structure in this data, it is not clear a priori how much better the more advanced methods will fair.”
Probabilistic programming with programmable inference (2018, methodology): “This case study can be viewed as an alternative implementation of the Automatic Bayesian Covariance Discovery (ABCD) approach from the Automated Statistican sys-tem [25].”
Differentiable Compositional Kernel Learning for Gaussian Processes (2018, methodology): “It can compactly approximate compositional kernel structures such as those used by the Automatic Statistician (Lloyd et al., 2014), but because the architecture is differentiable, it is end-to-end trainable with gradientbased optimization.”