A Comprehensive Survey of Neural Architecture Search

By Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao (Bernie) Huang, Zhihui Li, Xiaojiang Chen, Xin Wang. 2020

868 citations recorded by the source corpus.

Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers’ prior knowledge and experience. And due to the limitations of humans’ inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “Utilizing much more sample-efficient NAS and HPO methods (Turner et al., 2021; Ren et al., 2021) can translate directly into carbon footprint improvement.”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “NEURAL architecture search (NAS) has re7cently attracted massive interest from the deep learning community because experts do not have inordinate amounts of time and labor designing neural networks [13], [18], [29], [35], [42], [47], [50], [71].”
Dynamic Slimmable Network (2021, background): “In recent years, many different approaches have been proposed to improve the inference efficiency of neural networks, including network pruning [40, 47, 19, 20, 48, 50], weight quantization [30], knowledge distillation [2, 52, 21], manually and automatically designing of efficient networks [56, 53, ”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, background): “…design, with achieving remarkable success on image recognition (Tan & Le, 2019; Li et al., 2021b; 2020), language modeling (Jiang et al., 2019), and other deep learning ap-1 plications (Ren et al., 2020; Cheng et al., 2020; Chen et al., 2019b; Hu et al., 2021; Zhu et al., 2021; Ren et al., 2021).”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “As mentioned in §1, early NAS methods are built upon discrete encodings [42, 43, 12, 44, 45], which face the scalability challenge [46, 47] in large search spaces.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “TO date, deep learning [1], [2], [3], [4], [5] has substantially promoted the development of computer vision.”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “We refer to [54] for a more comprehensive NAS review.”
A Survey of Deep Active Learning (2020, methodology): “ query mechanism; however, the existing DL model is not necessarily optimal for the current DAL task. Active-iNAS [50] accordingly challenges this assumption and uses neural architecture search (NAS) [124] technology to dynamically search for the most effective model architectures while conducting a”
NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, background): “performance in specific search space with constrained resources [1], [2].”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “A typical NAS approach consists of three key components: a search space, a search strategy, and an architecture evaluation mechanism (White et al., 2023; Ren et al., 2021).”