Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data

By Patrick Schratz, Jannes Muenchow, Eugenia Iturritxa, Jakob Richter, Alexander Brenning. Ecological Modelling, 2018

439 citations recorded by the source corpus.

Abstract While the application of machine-learning algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages (such as R or Python), there are several practical challenges in the field of ecological modeling related to unbiased performance estimation. One is the influence of spatial autocorrelation in both hyperparameter tuning and performance estimation. Grouped cross-validation strategies have been proposed in recent years in environmental as well as medical contexts to reduce bias in predictive performance. In this study we show the effects of spatial autocorrelation on hyperparameter tuning and performance estimation by comparing several widely used machine-learning algorithms such as boosted regression trees (BRT), k-nearest neighbor (KNN), random forest (RF) and support vector machine (SVM) with traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM) in terms of predictive performance. Spatial and non-spatial cross-validation methods were used to evaluate model performances aiming to obtain bias-reduced performance estimates. A detailed analysis on the sensitivity of hyperparameter tuning when using different resampling methods (spatial/non-spatial) was performed. As a case study the spatial distribution of forest disease ( Diplodia sapinea ) in the Basque Country (Spain) was investigated using common environmental variables such as temperature, precipitation, soil and lithology as predictors. Random Forest (mean Brier score estimate of 0.166) outperformed all other methods with regard to predictive accuracy. Though the sensitivity to hyperparameter tuning differed between the ML algorithms, there were in most cases no substantial differences between spatial and non-spatial partitioning for hyperparameter tuning. However, spatial hyperparameter tuning maintains consistency with spatial estimation of classifier performance and should be favored over non-spatial hyperparameter optimization. High performance differences (up to 47%) between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) cross-validation settings showed the high need to account for the influence of spatial autocorrelation. Overoptimistic performance estimates may lead to false actions in ecological decision making based on biased model predictions.

Subfields: Economics and Econometrics, Environmental Engineering, Genetics

View Paper: https://www.semanticscholar.org/paper/b271d287baff4f0a34feae1d146579f3b0ae0514

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Environmental Engineering

In Genetics

Hanna Meyer

Citations in the corpus

Machine learning for digital soil mapping: Applications, challenges and suggested solutions (2020, methodology): “Schratz et al. (2019) show that hyperparameter tuning is also affected by spatial autocorrelation, and that overoptimistic results are reported when the same data are used for performance assessment and parameter tuning.”
Mapping dominant leaf type based on combined Sentinel-1/-2 data – Challenges for mountainous countries (2021, positive): “Such a reference data stratification strategy enables to reduce bias in training, validation and testing and, thus, leads to a better evaluation of model performances (Roberts et al., 2017; Schratz et al., 2019).”
Predicting into unknown space? Estimating the area of applicability of spatial prediction models (2020, methodology): “…relevance of accounting for spatial dependencies for reliable performance estimation via cross-19 validation has been recently highlighted by many studies (Ploton et al. 2020, Brenning 2012, Roberts et al. 2017,20 Schratz et al. 2019, Meyer et al. 2018, Valavi et al. 2018, Pohjankukka et al. 2017).”
Importance of spatial predictor variable selection in machine learning applications - Moving from data reproduction to spatial prediction (2019, background): “The results of this study strongly suggest that spatial cross-validation needs to be considered not only for model validation and model tuning (see Schratz et al., 2019, for a study on the relevance of spatial validation for hyperparameter tuning in machine learning applications) but also for…”