Hierarchical Representations for Efficient Architecture Search

By Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu. 2017

955 citations recorded by the source corpus.

We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Neural Architecture Search (NAS) [42] and neural evolution [1, 22, 26, 30, 34] can search network architectures from a gigantic search space.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Subsequent work accelerated training efficiency and the efficacy of the procedure [30, 38, 28, 29], eventually making the method amenable to a unified optimization based on a differentiable process [30].”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “Given a search space of TensorFlow programs, we use evolution [11, 12, 13, 14, 15, 16, 17] to search for models that achieve as low of a validation loss as possible given a fixed amount of training compute.”
Squeeze-and-Excitation Networks (2017, background): “Recently, compositions which have been learned in an automated manner [26, 54, 55] have shown competitive performance.”
Progressive Neural Architecture Search (2017, methodology): “PNAS also outperforms the Hierarchical EA method of [18], while using 36 times less compute.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Improved hardware now allows scaling up evolution to produce high-quality image classifiers (Real et al. 2017; Xie and Yuille 2017; Liu et al. 2018b).”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “, 2017), and using hierarchical representation of architectures (Liu et al., 2018).”
DARTS: Differentiable Architecture Search (2018, methodology): “…approaches as it does not involve any controllers (Zoph and Le, 2016; Baker et al., 2016; Zoph et al., 2017; Pham et al., 2018b), hypernetworks (Brock et al., 2017) or performance predictors (Liu et al., 2017a), yet it is generic enough to search for both convolutional and recurrent architectures.”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “Inspired by recent work [35, 36, 25, 20], we use a reinforcement learning approach to find Pareto optimal solutions for our multi-objective search problem.”
AutoML: A Survey of the State-of-the-Art (2019, background): “The common structures includes entire structure [5]–[7], cell-based structure [6], [8]– [11], hierarchical structure [12] and morphism-based structure [13], etc.”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “Convolutions are applied in the order of ReLU-Conv-BN, and the depthwise separable convolution is always applied twice (Zoph et al., 2017; Real et al., 2018; Liu et al., 2017a; 2019).”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, neutral): “8 3150 evolution Hierarchical evolution [16] 3.”
Weight Agnostic Neural Networks (2019, methodology): “In order to achieve state-of-the-art results, recent methods narrow the search space to architectures composed of basic building blocks with strong domain priors such as CNNs [70, 78, 95, 128], recurrent cells [53, 78, 128] and self-attention [107].”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “…promising approach for speeding up performance estimation, which treats all architectures as different subgraphs of a supergraph (the one-shot model) and shares weights between architectures Saxena & Verbeek (2016); Brock et al. (2017); Pham et al. (2018); Liu et al. (2018b); Bender et al. (2018).”
A Comprehensive Survey of Neural Architecture Search (2020, background): “Although RS has proven to be a strong baseline [24, 33, 104, 127], relevant research is still insufficient.”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “For example, on the CIFAR10 dataset, the largescale evolutionary algorithm [12] consumed 11 days on 250 GPUs, and the hierarchical evolutionary algorithm [13] spent 1.”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “Earlier NAS methods [19,20,26,37,38] train thousands of candidate architectures from scratch (on a smaller proxy task) and use their validation performance as the feedback to an algorithm that learns to focus on the most promising regions in the search space.”
Real-Time Federated Evolutionary Neural Architecture Search (2020, methodology): “Recently, evolutionary approaches to NAS have received increasing attention [10]–[13], and hybrid methods that combine evolutionary search with the gradient method [14] have also been reported [15].”