On Optimal Generalizability in Parametric Learning
By Ahmad Beirami, Meisam Razaviyayn, Shahin Shahrampour, Vahid Tarokh. 2017
58 citations recorded by the source corpus.
We consider the parametric learning problem, where the objective of the learner is determined by a parametric loss function. Employing empirical risk minimization with possibly regularization, the inferred parameter vector will be biased toward the training samples. Such bias is measured by the cross validation procedure in practice where the data set is partitioned into a training set used for training and a validation set, which is not used in training and is left to measure the out-of-sample performance. A classical cross validation strategy is the leave-one-out cross validation (LOOCV) where one sample is left out for validation and training is done on the rest of the samples that are presented to the learner, and this process is repeated on all of the samples. LOOCV is rarely used in practice due to the high computational complexity. In this paper, we first develop a computationally efficient approximate LOOCV (ALOOCV) and provide theoretical guarantees for its performance. Then we use ALOOCV to provide an optimization algorithm for finding the regularizer in the empirical risk minimization framework. In our numerical experiments, we illustrate the accuracy and efficiency of ALOOCV as well as our proposed framework for the optimization of the regularizer.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank Hutter
- Thomas Brox
- Arber Zela
- Quoc V. Le
- Michael I. Jordan
- Percy Liang
- Sergey Levine
- Bernt Schiele
In Computational Mechanics
- Michael I. Jordan
- Thomas Elsken
- Arber Zela
- Thomas Brox
- Tonmoy Saikia
- Riccardo Grazzi
- Frank Hutter
- Paolo Frasconi
Citations in the corpus
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “y (Bengio, 2000; Pedregosa, 2016; Beirami et al., 2017). Another strategy is to approximate the inner problem with a dynamical system (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017; 2018), where the optimization dynamics could, e.g., describe gradient descent. In the case that the min”
- Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, methodology): “While a possible strategy (implicit differentiation) is to apply the implicit function theorem to ∇ L λ = 0 (Pedregosa, 2016; Koh and Liang, 2017; Beirami et al., 2017), another compelling approach is to replace the inner problem with a dynamical system.”
- A Swiss Army Infinitesimal Jackknife (2018, methodology): “A number of recent papers in machine learning have provided related linear approximations for the special case of leave-one-out cross-validation [Koh and Liang, 2017, Rad and Maleki, 2018, Beirami et al., 2017], though their analyses lack the generality of the statistical perspective.”