Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

By Arber Zela, Aaron Klein, Stefan Falkner, Frank Hutter. 2018

175 citations recorded by the source corpus.

While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “We conjecture that the relation between model configuration and hyperparameter choice (Zela et al., 2018) may entail the potential to formulate hyperparameter-scaling policies similar in nature to the model-scaling polices, and that these too fall under the scope of the form we find in this work.”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Few approaches have been developed for the joint searching of HAS [26, 54].”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Orthogonally, a few studies benefit from extending the search space to two such aspects simultaneously (Zela et al., 2018; Miikkulainen et al., 2019; Noy et al., 2019).”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “…one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or multi-ﬁdelity optimization [Baker et al., 2017b, Falkner et al., 2018, Li et al., 2017, Zela et al., 2018], however their applicability still often remains restricted to rather simple tasks and small datasets.”
Evaluating the Search Phase of Neural Architecture Search (2019, neutral): “This phenomenon was already noticed by Zela et al. (2018), and concurrently to our work by Li & Talwalkar (2019); Xie et al. (2019); Ying et al. (2019), but the analysis of its impact or its causes went no further.”
Optimized Generic Feature Learning for Few-shot Classification across Domains (2020, background): “The budget usually refers to the number of mini-batch updates or number of epochs [12, 47] but may also be other parameters such as image resolution or training time [60].”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, methodology): “While BOHB can, in principle, also be used to optimize architectural parameters [74, 60], its performance degrades compared to gradient-based approaches as the dimensionality of the search space grows.”
Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search (2020, methodology): “This undesirable effect on relative ranking is more prominent when the cheap approximation set-up is too dissimilar to the full evaluation [9].”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Few approaches have been developed for the joint searching of hyperparameter and architecture (Klein & Hutter, 2019; Zela et al., 2018).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Although it is common practice to optimize architecture and HPCs in sequence, recent evidence suggests that they should be jointly optimized (Lindauer & Hutter, 2020; Zela et al., 2018).”
Learnable Embedding Space for Efficient Neural Architecture Compression (2019, background): “Mendoza et al. (2016), Jenatton et al. (2017) and Zela et al. (2018) also fall into this category.”
Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “[60] Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter.”