Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search

By Yong Guo, Yaofo Chen, Yin Zheng, Peilin Zhao, Jian Chen, Junzhou Huang, Mingkui Tan. 2020

78 citations recorded by the source corpus.

Neural architecture search (NAS) has become an important approach to automatically find effective architectures. To cover all possible good architectures, we need to search in an extremely large search space with billions of candidate architectures. More critically, given a large search space, we may face a very challenging issue of space explosion. However, due to the limitation of computational resources, we can only sample a very small proportion of the architectures, which provides insufficient information for the training. As a result, existing methods may often produce suboptimal architectures. To alleviate this issue, we propose a curriculum search method that starts from a small search space and gradually incorporates the learned knowledge to guide the search in a large space. With the proposed search strategy, our Curriculum Neural Architecture Search (CNAS) method significantly improves the search efficiency and finds better architectures than existing NAS methods. Extensive experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Dai
2. Wenwu Zhu
3. Jian Chen
4. Xin Wang
5. Yong Guo
6. Xingang Pan
7. Christian Theobalt
8. Dahua Lin

In Computer Vision and Pattern Recognition

1. Bo Dai
2. Luc Van Gool
3. Chen Change Loy
4. Dahua Lin
5. Jian Chen
6. Xingang Pan
7. Christian Theobalt
8. Dacheng Tao

Citations in the corpus

• Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “optimal cell architecture [9]–[13] or a whole network architecture [14]–[17].”
• Sample-Efficient Neural Architecture Search by Learning Action Space (2019, unknown): “3 CNAS ([35]) c/o X 3.”
• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “Moreover, insights from curriculum learning [7,44,8,5,9,10,33] tell that training models in a meaningful order may ease the training on difficult tasks with model weights initialized on easy samples.”
• A Survey on Curriculum Learning (2021, methodology): “In addition to methodologies, more efforts should be made to explore the power of CL in more various applications, including both cutting-edge research areas (e.g., meta-learning, continual learning, NAS, graph neural network, self-supervised learning, etc.) and traditional machine learning topics (”