Simple And Efficient Architecture Search for Convolutional Neural Networks

By Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. 2017

245 citations recorded by the source corpus.

Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “Neural Architecture Search We give a short overview of the most relevant works in neural architecture search (NAS) and refer to Elsken et al. (2018) for a comprehensive survey on the topic.”
A Comprehensive Survey of Neural Architecture Search (2020, background): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “, employing network morphisms (Cai et al., 2018a;b; Elsken et al., 2017; 2019a), weight sharing within search models (Saxena & Verbeek, 2016; Bender et al.”
A Survey on Neural Architecture Search (2019, methodology): “Elsken et al. (2019) Elsken et al. (2019) propose another evolutionary algorithm which shares many common aspects with their earlier work (Elsken et al., 2018) (see Section 3.2).”
NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “The search algorithms mainly includes heuristic algorithm [19], [21], [32]–[34], reinforcement learning [35]–[38], [38], [39], the Bayesian optimization method [40], [41] and the gradient-based method [20], [42], [43].”
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (2018, methodology): “2 RELATED WORK The use of machine learning techniques, such as reinforcement learning or neuro-evolution, to replace human experts in designing neural network architectures, usually referred to as neural architecture search, has drawn an increasing interest (Zoph & Le, 2017; Liu et al., 2018a;b;c; C”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “…deep RL and evolution, different approaches have been applied to architecture search and hyperparameter optimization: bayesian optimization [15], cascade-correlation [14], boosting [9], deep-learning based tree search [30], hill-climbing [13], model-based approaches [21], and random search [4].”
Auto-Keras: An Efficient Neural Architecture Search System (2018, methodology): “Initial efforts have been devoted to making use of network morphism in neural architecture search [6, 11].”
Path-Level Network Transformation for Efficient Architecture Search (2018, background): “Therefore, automatic architecture design has recently drawn much attention (Zoph & Le, 2017; Zoph et al., 2017; Liu et al., 2017; Cai et al., 2018; Real et al., 2018; Pham et al., 2018; Elsken et al., 2017).”
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search (2018, methodology): “ling approach that generates models with state-of-the-art accuracy. This approach may be ad-hoc as no analysis is presented on how the progressive search affects the trade-off frontier. Elsken et al. [11] use a simple hill climbing method along with a network morphism [46] approach to optimize netwo”
Demystifying Parallel and Distributed Deep Learning (2018, background): “6,94,133,165,172] Evolutionary[169,192,234,242] Hyper-Parameter Search[13,89,91,118,131,163,169,219,256] Architecture Search: Reinforcement[12,193,264,268,269], Evolutionary[160,203,204,242,251], SMBO[27,71,159,161,176] Fig. 18. Overview of Distributed Deep Learning Methods Centralization (§7.2) Com”
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search (2019, methodology): “These algorithms are widely used in NAS [2, 19, 5, 9].”
Efficient Neural Architecture Search with Network Morphism (2018, methodology): “Another simple approach [8] is to use random algorithm and hill-climbing, which can only explore the neighborhoods of the searched area each time, and could”
Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “A recent focus in NAS research has been on computational efficiency with algorithms utilizing gradient-based optimization [8, 12, 36, 55, 59, 56], network morphisms [7, 13, 14], or multi-fidelity optimization [3, 17, 60].”
Understanding and Simplifying One-Shot Architecture Search (2018, unknown): “Copyright 2018 by the author(s). proposed (Brock et al., 2017; Pham et al., 2018; Elsken et al., 2017; Liu et al., 2017a; Cai et al., 2017; Liu et al., 2017b).”
Auto-Keras: Efficient Neural Architecture Search with Network Morphism (2018, methodology): “There are four network morphism operations we could per- form on a neural network f ∈ F (Elsken et al., 2017), which can all be reflected in the change of the computational graph G.”
A Survey on Evolutionary Neural Architecture Search (2020, methodology): “For example, in [32] the evolutionary operators only contain mutation and no crossover, and the selection strategy is relatively simple.”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, methodology): “Network morphisms [12, 14, 23, 25] make use of the previous learned weights to initialize new candidate architectures, thereby speeding up the performance estimation procedure.”