Fast cross-validation via sequential testing

By Tammo Krueger, Danny Panknin, Mikio L. Braun. 2012

55 citations recorded by the source corpus.

With the increasing size of today's data sets, nding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which uses nonparametric testing coupled with sequential analysis to determine the best parameter set on linearly increasing subsets of the data. By eliminating underperforming candidates quickly and keeping promising candidates as long as possible, the method speeds up the computation while preserving the power of the full cross-validation. Theoretical considerations underline the statistical power of our procedure. The experimental evaluation shows that our method reduces the computation time by a factor of up to 120 compared to a full cross-validation with a negligible impact on the accuracy.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Frank Hutter
2. Ameet Talwalkar
3. Aaron Klein
4. Stefan Falkner
5. B. Recht
6. Quoc V. Le
7. Liam Li
8. Moritz Hardt

In Control and Systems Engineering

1. Philipp Hennig
2. Frank Hutter
3. Aaron Klein
4. Ameet Talwalkar
5. Moritz Hardt
6. Lisha Li
7. Afshin Rostamizadeh
8. Giulia DeSalvo

Citations in the corpus

• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “Krueger et al. [16] showed that in practical applications small subsets can suffice to estimate a configuration’s quality, and proposed a cross-validation scheme that sequentially tests a fixed set of configurations on a growing subset of the data, discarding poorly-performing configurations early.”
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Krueger et al. (2015) proposes a heuristic based on sequential analysis to determine stopping times for training configurations on increasing subsets of the data.”
• A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “(Krueger et al., 2015) relies on a heuristic early-stopping rule based on sequential analysis to terminate poor configurations.”
• Fast Bayesian hyperparameter optimization on large datasets (2017, background): “Krueger et al. (2015) showed that, in practical applications, small subsets can suffice to estimate a configuration’s quality, and proposed a cross-validation scheme that sequentially tests a fixed set of configurations on a growing subset of the data, discarding poorly-performing configurations early.”
• Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits (2016, methodology): “Krueger et al. (2015) instead proposes a heuristic based on sequential analysis to determine stopping times for training configurations on increasing subsets of the data.”
• Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization (2016, unknown): “One recent work of particular interest proposes a heuristic based on sequential analysis to determine stopping times for training configurations on increasing subsets of the data (Krueger et al., 2015).”