Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation

By Bernd Bischl, Olaf Mersmann, Heike Trautmann, Claus Weihs. 2012

165 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Visualizing the Feature Importance for Black Box Models (2018, methodology): “ Dby GEd(f;^ D) = 1 n P n i=1 L(f^(x(i));y(i)): (1) A better estimate for the generalization error of an ML algorithm can be obtained using resampling techniques such as cross-validation or bootstrap [1]. PD Plots [10] visualize the marginal relationship between features of interest and the expected”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, background): “…2015; Tsamardinos et al., 2018), but while such model selection strategies are an important area of research, houldout or K-fold cross-validation remain the most prominent choices (Henery, 1994; Kohavi and John, 1995; Hastie et al., 2001; Guyon et al., 2010; Bischl et al., 2012; Raschka, 2018).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “An overview of existing resampling techniques can be found in Bischl et al. (2012) or Boulesteix et al. (2008).”
Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization (2021, methodology): “An alternative solution [7] involves the use of hyperparameter-tuning strategies that are data dependent, which try to minimize the expected generalization error of the algorithm over a hyperparameter search space of considered candidate configurations, usually by evaluating predictions on”
Recent advances and applications of surrogate models for finite element method computations: a review (2022, methodology): “More details on various resampling approaches that are used for validation of surrogate models are discussed by Bischl et al. (2012).”
TSE-IDS: A Two-Stage Classifier Ensemble for Intelligent Anomaly-Based Intrusion Detection System (2019, methodology): “Subsampling is very similar to classical bootstrap [61].”
Tunability: Importance of Hyperparameters of Machine Learning Algorithms (2018, methodology): “…which try to minimize the expected generalization error of the inducing algorithm over a hyperparameter search space of considered candidate conﬁgurations, usually by evaluating predictions on an independent test set, or by running a resampling scheme such as cross-validation (Bischl et al., 2012).”