A Survey on Neural Architecture Search
By Martin Wistuba, Ambrish Rawat, Tejaswini Pedapati. 2019
293 citations recorded by the source corpus.
The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search. The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements. However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge. Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners. With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches. We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models. Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Neural Architecture Search & AutoML
Citations in the corpus
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Previous related surveys classified existing work mainly according to the basic components of NAS: search space, search strategy and evaluation strategy [26, 27].”
- Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, background): “Recent surveys on architecture search can be found in [39, 40].”
- Evolutionary Architecture Search for Graph Neural Networks (2020, background): “Most of existing works can be roughly classified into three categories, including reinforcement learning, Bayesian, and evolutionary optimizations [12], [27].”
- Optimization Models for Machine Learning: A Survey (2019, background): “For that, in addition to publications in classical operations research journals, this paper surveys machine learning and artificial intelligence conferences and journals, such as the conference on Association for the Advancement of Artificial Intelligence and the International Conference on Machine ”
- Delphi: A Cryptographic Inference Service for Neural Networks (2020, unknown): “The insight behind our planner is to adapt techniques for neural architecture search (NAS) and hyperparameter optimization (see [7, 25] for in-depth surveys of these areas) to our setting.”
- A Survey on Evolutionary Neural Architecture Search (2020, methodology): “[27] followed these three stages with an additional review about the multiple objectives in NAS.”
- Techniques for Automated Machine Learning (2019, methodology): “Several surveys on AutoMHT [101], AutoDL [24; 94; 37], and general AutoML [72] have been published from different perspectives, unveiling the mysterious art behind AutoML.”
- Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “Deep model compression techniques such as pruning [201], quantization [201], lowrank approximation [202], knowledge distillation [203], and neural architecture search [204] to new a few, is an area”