Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

By Aaron Klein, Frank Hutter. 2019

106 citations recorded by the source corpus.

Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate a better empirical evaluation of HPO methods by providing benchmarks that are cheap to evaluate, but still represent realistic use cases. We believe these benchmarks provide an easy and efficient way to conduct reproducible experiments for neural hyperparameter search. Our benchmarks consist of a large grid of configurations of a feed forward neural network on four different regression datasets including architectural hyperparameters and hyperparameters concerning the training pipeline. Based on this data, we performed an in-depth analysis to gain a better understanding of the properties of the optimization problem, as well as of the importance of different types of hyperparameters. Second, we exhaustively compared various different state-of-the-art methods from the hyperparameter optimization literature on these benchmarks in terms of performance and robustness.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Evaluation & Benchmarks

In Neural Architecture Search & AutoML

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “NAS benchmarks Two earliest benchmarks were released during 2019: NAS-Bench-101 [45] and NAS-HPO-Bench [24].”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Few approaches have been developed for the joint searching of HAS [26, 54].”
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “In response to this problem, NAS-Bench-101 (Ying et al., 2019) and NAS-HPO-Bench (Klein & Hutter, 2019) are proposed.”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Few approaches have been developed for the joint searching of hyperparameter and architecture (Klein & Hutter, 2019; Zela et al., 2018).”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, background): “…et al., 2018), Dragonﬂy (Kandasamy et al., 2020) and BOHB (Falkner et al., 2018) on the surrogate benchmark for HPO on DNNs on the letter dataset (Falkner et al., 2018), a joint HPO+NAS benchmark (Klein and Hutter, 2019) on the Naval-Propulsion dataset and a pure NAS benchmark (Zela et al., 2020).”
Supplementary material for DEHB: Evolutionary Hyperband for Scalable, Robust and Efﬁcient Hyperparameter Optimization (2021, methodology): “To facilitate HPO research involving feed-forward neural networks, [Klein and Hutter, 2019] introduced NAS-HPO-Bench with a search space composed of hyperparameters that parameterize the architecture of a 2-layer feed-forward network 6 , along with hyperparameters for its training procedure.”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, methodology): “The last set of experiments we conduct consist of tuning the HPs of a 2-layer feed-forward neural network on 4 data sets [25], namely { slice localization, protein structure, naval propulsion, parkinsons telemonitoring }.”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “[27], [60], [62] have tried to address this problem by building benchmark datasets for NAS.”
Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance (2023, methodology): “The characteristics of each benchmark function are available in Table 11 For the HPO benchmarks, we used HPOBench (Eggensperger et al., 2021) in Table 12, HPOlib (Klein & Hutter, 2019) in Table 13, and JAHS-Bench-201 (Bansal et al., 2022) in Table 14.”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “NAS-HPO-Bench [19] evaluated 62208 configurations in the joint NAS and hyper-parameter space for a simple 2-layer”