Practical Deep Learning Architecture Optimization

By Martin Wistuba. 2017

21 citations recorded by the source corpus.

The design of neural network architectures for a new data set is a laborious task which requires human deep learning expertise. In order to make deep learning available for a broader audience, automated methods for finding a neural network architecture are vital. Recently proposed methods can already achieve human expert level performances. However, these methods have run times of months or even years of GPU computing time, ignoring hardware constraints as faced by many researchers and companies. We propose the use of Monte Carlo planning in combination with two different UCT (upper confidence bound applied to trees) derivations to search for network architectures. We adapt the UCT algorithm to the needs of network architecture search by proposing two ways of sharing information between different branches of the search tree. In an empirical study we are able to demonstrate that this method is able to find competitive networks for MNIST, SVHN and CIFAR-10 in just a single GPU day. Extending the search time to five GPU days, we are able to outperform man-made architectures and our competitors which consider the same types of layers.

Subfields: Computer Vision and Pattern Recognition, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Tejaswini Pedapati
2. Ambrish Rawat
3. Martin Wistuba

In Materials Chemistry

1. Tejaswini Pedapati
2. Ambrish Rawat

Citations in the corpus

• DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, neutral): “Therefore, NAS is always formulated as a hyper-parameter optimization problem, whose algorithmic realization spans evolution algorithm [21, 7], reinforcement learning [27], Bayesian optimization [9], Monte Carlo Tree Search [24], and differentiable architecture search [14, 26, 3].”
• A Survey on Neural Architecture Search (2019, methodology): “Wistuba (2018a) In a similar line of work, Wistuba (2018a) also utilizes functionpreserving mutations but uses a cell-based search space.”
• Deep Learning Architecture Search by Neuro-Cell-Based Evolution with Function-Preserving Mutations (2018, methodology): “Other methods that try to optimize neural network architectures or their hyperparameters are based on model-based optimization [7, 14, 22, 26], random search [2] and Monte-Carlo Tree Search [19,27].”