Efficient Progressive Neural Architecture Search

By Juan-Manuel Pérez-Rúa, Moez Baccouche, Stéphane Pateux. 2018

44 citations recorded by the source corpus.

This paper addresses the difficult problem of finding an optimal neural architecture design for a given image classification task. We propose a method that aggregates two main results of the previous state-of-the-art in neural architecture search. These are, appealing to the strong sampling efficiency of a search scheme based on sequential model-based optimization (SMBO), and increasing training efficiency by sharing weights among sampled architectures. Sequential search has previously demonstrated its capabilities to find state-of-the-art neural architectures for image classification. However, its computational cost remains high, even unreachable under modest computational settings. Affording SMBO with weight-sharing alleviates this problem. On the other hand, progressive search with SMBO is inherently greedy, as it leverages a learned surrogate function to predict the validation error of neural architectures. This prediction is directly used to rank the sampled neural architectures. We propose to attenuate the greediness of the original SMBO method by relaxing the role of the surrogate function so it predicts architecture sampling probability instead. We demonstrate with experiments on the CIFAR-10 dataset that our method, denominated Efficient progressive neural architecture search (EPNAS), leads to increased search efficiency, while retaining competitiveness of found architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Wei-Wei Tu
3. Quanming Yao
4. Yang Yu
5. Dahua Lin
6. Junjie Yan
7. Yi-Qi Hu
8. Mathieu Salzmann

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Junjie Yan
5. Yi Yang
6. Chunhua Shen
7. Ming-Hsuan Yang
8. Dahua Lin

In Neural Architecture Search & AutoML

1. Dahua Lin
2. Junjie Yan
3. Zhaobai Zhong
4. Yang Yu
5. Qiang Yang
6. Hugo Jair Escalante
7. Isabelle M Guyon
8. Yu-Feng Li

Citations in the corpus

• Evaluating the Search Phase of Neural Architecture Search (2019, background): “2 Related Work Since its introduction in [36], neural architecture search demonstrated great potential to surpass the human design of deep networks for both visual recognition [17, 1, 7, 26, 15] and natural language processing [36, 27, 20, 37, 17, 5].”
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, methodology): “As RL is slow to converge, to make the search faster, transfer learning, which is firstly used to cut the search space into several blocks, was developed in [23], [64]; then, a parameter sharing scheming is proposed in [116] to further narrow down the search space; and greedy search has also been co”
• IRLAS: Inverse Reinforcement Learning for Architecture Search (2018, methodology): “Along this direction, a number of network search methods have been developed, including evolution [26, 31], surrogate model based search [18, 21], and reinforcement learning [36, 37, 34, 4].”
• When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks (2019, unknown): “Other efforts [19, 26, 2] utilize the weight sharing mechanism to reduce the costs of evaluating each searched candidate architecture.”
• Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, unknown): “These works have been aggregated by EPNAS [29].”