SNAS: Stochastic Neural Architecture Search
By Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin. International Conference on Learning Representations, 2018
1,014 citations recorded by the source corpus.
We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline. In this work, NAS is reformulated as an optimization problem on parameters of a joint distribution for the search space in a cell. To leverage the gradient information in generic differentiable loss for architecture search, a novel search gradient is proposed. We prove that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable reward to enforce a resource-efficient constraint. In experiments on CIFAR-10, SNAS takes less epochs to find a cell architecture with state-of-the-art accuracy than non-differentiable evolution-based and reinforcement-learning-based NAS, which is also transferable to ImageNet. It is also shown that child networks of SNAS can maintain the validation accuracy in searching, with which attention-based NAS requires parameter retraining to compete, exhibiting potentials to stride towards efficient NAS on big datasets. We have released our implementation at this https URL.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
View Paper: https://arxiv.org/abs/1812.09926
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Dacheng Tao
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Sergey Levine
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Alan L. Yuille
- Jitendra Malik
In Neural Architecture Search & AutoML
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
- MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “The third method is gradient-based [17, 28, 5, 19, 4], which relaxes the categorical design choices to continuous variables and then leverages the efficient gradient backpropagation so that it can finish searching within as little as several GPU days.”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “This strategy has an over-fitting problem, which is also found in NAS [11, 52, 12, 50].”
- FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “One category of NAS is differentiable neural architecture search (DNAS).”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “To this end, SNAS [45] starts with the delayed reward of reinforcement learning, and analyzes the reason why delayed reward leads to the slow convergence speed of reinforcement learning when performing architecture search.”
- DADA: Differentiable Automatic Data Augmentation (2020, neutral): “In the field of network architecture search (NAS), SNAS [26] and GDAS [5] use this estimator and achieved good performance.”
- AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “DARTS[17] and SNAS[31] regard the connection as a weight via back-propagation to optimize.”
- Semi-Supervised Neural Architecture Search (2020, background): “ the highest computational cost in NAS. Popular methods usually consume hundreds to thousands of GPU days to discover eventually good architectures [41, 20, 17]. To address this problem, one-shot NAS [2, 18, 15, 35] uses a supernet to include all candidate architectures via weight sharing and trains”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Therefore, NAS is always formulated as a hyper-parameter optimization problem, whose algorithmic realization spans evolution algorithm [21, 7], reinforcement learning [27], Bayesian optimization [9], Monte Carlo Tree Search [24], and differentiable architecture search [14, 26, 3].”
- NAS evaluation is frustratingly hard (2019, methodology): “As such, future research could investigate the optimal wiring at a global level: an interesting work in this direction is Xie et al. (2019a).”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “NAS Algorithm Source Test error Queries Runtime Method Avg Best SNAS [50] 2.”
- Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “We also proposed a simple early stopping criterion and adaptive regularization for DARTS which substantially improve our understanding of DARTS’ failure modes and lead to much more robust versions across many settings and tasks.”
- DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “However, the original DARTS fails to search on CIFAR100 as the selected architecture is full of skip-connect s [4], and most previous works on differentiable search [4, 20, 34] do not search on ImageNet.”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “, 2018), and later these methods were generalized into a more elegant framework named one-shot architecture search (Brock et al., 2018; Cai et al., 2019; Liu et al., 2019; Pham et al., 2018; Xie et al., 2019), in which an over-parameterized network or super-network covering all candidate operations ”
- Searching for a Robust Neural Architecture in Four GPU Hours (2019, methodology): “[39] applied a similar technique to relax the discrete candidate sampling as ours.”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “Search Space Our setup follows DARTS and SNAS, where convolutional cells of 7 nodes are stacked for multiple times to form a network.”
- A Survey on Neural Architecture Search (2019, background): “The work by Xie et al. (2019b) builds upon the work of Liu et al. (2019b), but has three major differences.”