Generalized additive models for large data sets

By Simon N. Wood, Yannig Goude, S. Shaw. 2015

346 citations recorded by the source corpus.

We consider an application in electricity grid load prediction, where generalized additive models are appropriate, but where the data set's size can make their use practically intractable with existing methods. We therefore develop practical generalized additive model fitting methods for large data sets in the case in which the smooth terms in the model are represented by using penalized regression splines. The methods use iterative update schemes to obtain factors of the model matrix while requiring only subblocks of the model matrix to be computed at any one time. We show that efficient smoothing parameter estimation can be carried out in a well‐justified manner. The grid load prediction problem requires updates of the model fit, as new data become available, and some means for dealing with residual auto‐correlation in grid load. Methods are provided for these problems and parallel implementation is covered. The methods allow estimation of generalized additive models for large data sets by using modest computer hardware, and the grid load prediction problem illustrates the utility of reduced rank spline smoothing methods for dealing with complex modelling problems.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Statistics and Probability

Citations in the corpus

A brief introduction to mixed effects modelling and multi-model inference in ecology (2018, background): “Often, simple analyses will be sufficient (Murtaugh 2007), but more complex data structures often require more complex methods such as linear mixed effects models (Zuur et al 2009), generalized additive models (Wood et al 2015) or Bayesian inference (Ellison 2004).”
Scale-dependent interactions between tree canopy cover and impervious surfaces reduce daytime urban heat during summer (2019, methodology): “Models were fit using fast restricted maximum likelihood (52).”
Hierarchical generalized additive models in ecology: an introduction with mgcv (2019, methodology): “It then blocks the data and updates model ﬁ t within each block ( Wood, Goude & Shaw, 2015 ).”
Power systems data fusion based on belief propagation (2017, methodology): “The forecasts were computed using generalized additive models (GAMs) based on penalized iteratively re-weighted least squares method [22], [23].”
The cave of Shadows. Addressing the human factor with generalized additive mixed models (2015, methodology): “(3) Pinheiro and Bates (2000) and Ga lecki and Burzykowski (2013) provide extensive discussion of how autocorrelation processes can be accounted for within the mixed modeling framework; Wood et al. (2015) provides technical details for gamms.”
Twenty years of P-splines (2015, methodology): “Recently, and motivated by the need to improve the speed and stability of forecasting models, Wood et al. (2015) have developed efficient methods for fitting additive models to large data sets with correlated errors.”