Overfitting in Making Comparisons Between Variable Selection Methods

By Juha Reunanen. Journal of machine learning research, 2003

573 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/22ac71a2922a6478227f7670c81f465abc1e0366

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

k-Nearest Neighbour Classifiers (2007, background): “More sophisticated search strategies can be employed to better explore the search space; however, Reunanen [20] cautions that more intensive search strategies are more likely to overfit the training data.”
Comparison of Bayesian predictive methods for model selection (2015, methodology): “This variance in the estimate may lead to overﬁtting in the selection process causing nonoptimal model selection and inducing bias in the performance estimate for the selected model (e.g. Ambroise and McLachlan, 2002; Reunanen, 2003; Cawley and Talbot, 2010).”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, unknown): “, 2019; de Jesús Rubio, 2009), wrapper methods (Kohavi & John, 1997; Pudil et al., 1994; Reunanen, 2003; Eshelman, 1991; Cordón et al., 2006; Meda-Campaña, 2018; de Jesús Rubio, 2017) and embedded methods (Langley, 1994; Mundra & Rajapakse, 2009; Romero & Sopena, 2008).”
Tree Alignment Based on Needleman-Wunsch Algorithm for Sensor Selection in Smart Homes (2017, unknown): “However, the wrapper-based approach is more computationally expensive than the filter-based approach and may run the risk of overfitting [4,12].”
k-Nearest Neighbour Classifiers - A Tutorial (2020, background): “More sophisticated search strategies can be employed to better explore the search space; however, Reunanen [38] cautions that more intensive search strategies are more likely to overfit the training data.”
A systematic review on overfitting control in shallow and deep neural networks (2021, methodology): “Biased predictions using different selections of variables (Reunanen 2003), 7.”
Identifying Statistical Bias in Dataset Replication (2020, background): “For example, evaluating on benchmarks alone may steer us towards models that adaptively overfit (Reunanen, 2003; Rao et al., 2008; Dwork et al., 2015) to the test set and do not generalize.”
Regression on manifolds: Estimation of the exterior derivative (2011, methodology): “Additionally, we suggest selecting parameters in a sequential manner; this is to reduce overfitting caused by testing too many models [42, 44, 45].”
Bias in error estimation when using cross-validation for model selection (2006, background): “[2], Ambroise and McLachlan [3] and Reunanen [4], this gives a very biased estimate of the true error; not much better than the resubstitution estimate.”
Markov blanket-embedded genetic algorithm for gene selection (2007, background): “This so called internal cross validation [8,47] has been demonstrated to suffer from selection bias [8] or overfitting [47] problem, especially on datasets of small sample size.”
mr2PSO: A maximum relevance minimum redundancy feature selection method based on swarm intelligence for support vector machine classification (2011, methodology): “The SVM, based on statistical learning theory and structural risk minimization, selects key data points as its support vectors, and uses these support vectors for prediction.”
Learning in high-dimensional multimedia data: the state of the art (2015, methodology): “Several publications [30, 69] have presented various definitions and measurements for the relevance of a variable.”
Complete Search for Feature Selection in Decision Trees (2019, methodology): “This implies that oversearching may incur in feature subsets that overfit the data, thus reinforcing the conclusions of Doak (1992); Quinlan and Cameron-Jones (1995); Reunanen (2003) that oversearching increases overfitting.”