Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks

By Tobias Hinz, Nicolás Navarro-Guerrero, Sven Magg, Stefan Wermter. 2018

111 citations recorded by the source corpus.

Most learning algorithms require the practitioner to manually set the values of many hyperparameters before the learning process can begin. However, with modern algorithms, the evaluation of a given hyperparameter setting can take a considerable amount of time and the search space is often very high-dimensional. We suggest using a lower-dimensional representation of the original data to quickly identify promising areas in the hyperparameter space. This information can then be used to initialize the optimization algorithm for the original, higher-dimensional data. We compare this approach with the standard procedure of optimizing the hyperparameters only on the original input. We perform experiments with various state-of-the-art hyperparameter optimization algorithms such as random search, the tree of parzen estimators (TPEs), sequential model-based algorithm configuration (SMAC), and a genetic algorithm (GA). Our experiments indicate that it is possible to speed up the optimization process by using lower-dimensional data representations at the beginning, while increasing the dimensionality of the input later in the optimization process. This is independent of the underlying optimization procedure, making the approach promising for many existing hyperparameter optimization algorithms.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Neural Architecture Search (2019, background): “ide a competitive solution (Wistuba and Pedapati, 2019). Other similar paradigms include searching on a subset of the data (Sabharwal et al., 2016) and searching on a down-scaled version of the data (Hinz et al., 2018). It is also a common practice to search for networks with few cell repetitions an”
Regularize, Expand and Compress: NonExpansive Continual Learning (2020, unknown): “REC is our initial work for overcoming catastrophic forgetting and we will speed-up the hyperparameter optimization [12] in our future work.”
Dropout vs. batch normalization: an empirical study of their impact to deep learning (2020, background): “In [6] the authors propose to start the training phase with a lower dimension version of the input data.”
Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks (2018, background): “deep neural networks, especially in complex models such as GANs, can be computationally intensive [60], [61].”