Sample-Efficient Neural Architecture Search by Learning Action Space

By Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, Yuandong Tian. 2019

48 citations recorded by the source corpus.

Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy). As a result, using manually designed action space to perform NAS often leads to sample-inefficient explorations of architectures and thus can be sub-optimal. In order to improve sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS) that learns the action space to recursively partition the architecture search space into regions, each with concentrated performance metrics (\emph{i.e.}, low variance). During the search phase, as different architecture search action sequences lead to regions of different performance, the search efficiency can be significantly improved by biasing towards the regions with good performance. On the largest NAS dataset NasBench-101, our experimental results demonstrated that LaNAS is 22x, 14.6x and 12.4x more sample-efficient than random search, regularized evolution, and Monte Carlo Tree Search (MCTS) respectively. When applied to the open domain, LaNAS finds an architecture that achieves SoTA 98.0% accuracy on CIFAR-10 and 75.0% top1 accuracy on ImageNet (mobile setting), after exploring only 6,000 architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Colin White
2. Yash Savani
3. Xia Hu
4. Qingquan Song
5. Willie Neiswanger
6. Zhenguo Li
7. James T. Kwok
8. Hang Xu

In Computer Vision and Pattern Recognition

1. Colin White
2. Yash Savani
3. Zhenguo Li
4. Hang Xu
5. Xia Hu
6. Willie Neiswanger
7. Yuandong Tian
8. Qingquan Song

Citations in the corpus

• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “[61] Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian.”
• vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training (2022, unknown): “, neural architecture search [45], [55], [57], [62]).”
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, methodology): “In sample-based NAS algorithms, an architecture is selected in each iteration for full training [25, 33, 35, 42], and is computationally expensive.”
• Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction (2020, methodology): “The selected NAS methods include random search, reinforcement learning based search (RL), and a sample-efficient tree-based search: latent action neural architecture search (LaNAS) [29].”