Learning hyperparameter optimization initializations

By Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme. 2015

108 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Warm-starting can also be used for many aspects of AutoML, yet most research focuses on model selection and tuning (Gomes et al., 2012; De Miranda et al., 2012; Reif et al., 2012; Feurer et al., 2015a, 2015b; Wistuba et al., 2015b; Lindauer & Hutter, 2018).”
Survey on Automated Machine Learning (2019, methodology): “Meta-learning can be used to filter empirically bad performing candidate configurations (Alia and Smith-Miles, 2006; Wistuba et al., 2015a).”
Meta-Learning: A Survey (2018, neutral): “A very diﬀerent approach to the one above is to ﬁrst ﬁt a diﬀerentiable function f j ( θ i ) = P i,j on all prior evaluations of a speciﬁc task t j , and then use gradient descent to ﬁnd an optimized conﬁguration θ ∗ j per prior task [185].”
Warmstarting of Model-based Algorithm Configuration (2017, background): “Wistuba, Schilling, and Schmidt-Thieme (2015) proposed another approach for warmstarting the initial de-sign which does not depend on instance features and is not limited to conﬁgurations returned in previous optimization experiments.”
Recent Advances in Bayesian Optimization (2022, background): “To achieve this, Feurer et al. [70] introduced a negative Spearman correlation coefficient to measure the similarity between different datasets, while Wistuba et al. [262] identified the initial hyperparameter configurations via optimising a meta-loss.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “…” values that work well under many conditions, by tuning ranges and HPC prior distributions based on extensive meta-data (Feurer, Klein, et al., 2015a; Gijsbers et al., 2021; Perrone et al., 2019; Pfisterer et al., 2021; Probst et al., 2019; Van Rijn & Hutter, 2018; Wistuba et al., 2015a, 2015b).”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “rations to initialize search. Most work of this kind focus on hyper-parameter tuning, with particle swarm optimization [141], [155], evolutionary algorithm [142], or Bayesian optimization [7], [143], [166], [167]. Search space reﬁning: Meta-learning can accelerate conﬁgration search by reﬁning the s”
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization (2017, methodology): “Therefore, we propose an alternative using a pairwise hyperparameter performance comparison (Leite et al. 2012; Wistuba et al. 2015).”
Amazon SageMaker Autopilot: a white box AutoML solution at scale (2020, methodology): “We adopt the zero-shot learning approach ﬁrst introduced in Wistuba et al. (2015a;b) and aim to ﬁnd a set of k = 10 strategies such that, for a given customer’s dataset, at least one of them performs well.”
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates (2017, background): “In fact, several researchers have already applied this approach to simplify experiments (Birattari et al 2002; Snoek et al 2012; Bardenet et al 2014; Feurer et al 2015b; Wistuba et al 2015).”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “We also include results for F1-score and ROC AUC in Appendix D. Similar to prior work [20, 62], whenever we average across datasets, we use the average distance to the minimum (ADTM) metric, which consists of 0-1 scaling (after selecting the best hyperparameters, which helps protect against…”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “To aggregate results across datasets of varying difficulty, we use a metric similar to the distance to the minimum (or average distance to the minimum –ADTM– when averaged across datasets), used in Feurer et al. [2021] and introduced in Wistuba et al. [2015].”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “Extending these portfolio strategies which are learned offline, there are online portfolios which can select from a fixed set of machine learning pipelines, taking previous evaluations into account (Leite et al., 2013; Wistuba et al., 2015a,b; Fusi et al., 2018; Yang et al., 2019, 2020).”
Automatic Frankensteining: Creating Complex Ensembles Autonomously (2017, methodology): “∗Information Systems and Machine Learning Lab, University of Hildesheim, Germany methods from meta-learning to transfer knowledge from observed data sets to new ones to initialize the search [19].”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “To aggregate results across datasets of varying difﬁculty, we use a metric similar to the distance to the minimum (or average distance to the minimum (ADTM) when averaged across datasets), used in Feurer et al. [2021] and introduced in Wistuba et al. [2015].”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, background): “Finally, rather than ﬁtting a surrogate model to all past data, some transfer can be achieved by warm-starting BO with the solutions to the previous BO problems [10, 50].”
A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, methodology): “In each iteration of the sequential model-based optimization process, the first method (Wistuba et al. 2015a) uses knowledge extracted from experimental results on previous data sets to prune regions of the hyper-parameter space unlikely to contain good combinations of hyper-parameter values.”