Preventing Over-Fitting during Model Selection via Bayesian Regularisation of the Hyper-Parameters

By Gavin C. Cawley, Nicola L. C. Talbot. 2007

270 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning kernel logistic regression in the presence of class label noise (2014, methodology): “We then propose a robust Multiple Kernel Logistic Regression algorithm (rMKLR) based on the so-called Multiple Kernel Learning (MKL) framework (an extensive survey in recent advances of MKL is given in [16]) and the Bayesian regularisation technique [9] to automate the model selection step without u”
Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes (2017, methodology): “Adapting the Prior via Risk-based Empirical Bayes In order to avoid overfitting to the factual outcomes Y, we evaluate the empirical error in factual outcomes via leave-one-out cross-validation (LOO-CV) with Bayesian regularization [24]; the regularized objective function is thus given by R̂(θ;D) = ”
Deep Learning for Land Use and Land Cover Classification Based on Hyperspectral and Multispectral Earth Observation Data: A Review (2020, background): “The regularisation parameter in SVM plays a critical role in its performance; with well-tuned regularisations, SVMs tend to be resistant to overfitting and do not have any inherent problem when the number of observations is less than the number of attributes [93,94].”
Stability and Hypothesis Transfer Learning (2013, methodology): “As a useful consequence , its LOO prediction function is expressed in closed form as well, allowing a very efficient model selection (Cawley & Talbot, 2007).”
Evaluating the Unloading Gradient Pressure in Continuous Gas-lift Systems During Petroleum Production Operations (2014, methodology): “…where each output yi ∈ and the input xi ∈, LSSVM for regression problem is introduced as minimization of following formula (Suykens et al., 2002; Cawley and Talbot, 2007): min J (w, ξ ) = 1 − 2w2 + 1 2μ m∑ i=1 (ξi) 2 (1) s.t. : yi = wT ϕ (xi) + b + ξi, i = 1, 2, . . . , m where ϕ(xi) is a…”
A systematic review on overfitting control in shallow and deep neural networks (2021, background): “Nevertheless, a great number of parameters cause overfitting (Cawley and Talbot 2007; Tzafestas et al. 1996).”
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation (2010, methodology): “An attractive feature of kernel ridge regression is that it is possible to perform leave-one-out cross-validation in closed form, with minimal cost as a by-product of the training algorithm (Cawley and Talbot, 2003).”
Analysis of the AutoML Challenge Series 2015-2018 (2019, methodology): “The method was improved by adding a regularization of J2 (Cawley and Talbot, 2007).”
Model Selection: Beyond the Bayesian/Frequentist Divide (2010, methodology): “Such virtual leave-one-out estimators, requiring training a single classifier on all the data (see e.g., Cawley and Talbot, 2007a; Debruyne et al., 2 –8, in the collection of papers we have assembled).”
Design of the 2015 ChaLearn AutoML challenge (2015, methodology): “The authors perfected this method by adding a regularization of J2 in their subsequent work [48].”
Sparse Signal Recovery With Temporally Correlated Source Vectors Using Sparse Bayesian Learning (2011, background): “in overfitting [ 46 ], [47] due to limited data and too many parameters.”
Label-Noise Robust Logistic Regression and Its Applications (2012, result): “Bayesian regularisation was found comparable in performance to cross-validation [5], and in particular it was also demonstrated to be effective for L1-regularised logistic regression [4].”