You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

By Xinbang Zhang, Zehao Huang, Naiyan Wang, Shiming Xiang, Chunhong Pan. 2018

100 citations recorded by the source corpus.

Recently neural architecture search (NAS) has raised great interest in both academia and industry. However, it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a direct sparse optimization NAS (DSO-NAS) method. The motivation behind DSO-NAS is to address the task in the view of model pruning. To achieve this goal, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, sparse regularizations are imposed to prune useless connections in the architecture. Lastly, an efficient and theoretically sound optimization method is derived to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore it can be directly applied to large datasets like ImageNet and tasks beyond classification. Particularly, on the CIFAR-10 dataset, DSO-NAS achieves an average test error 2.74 percent, while on the ImageNet dataset DSO-NAS achieves 25.4 percent test error under 600M FLOPs with 8 GPUs in 18 hours. As for semantic segmentation task, DSO-NAS also achieve competitive result compared with manually designed architectures on the PASCAL VOC dataset. Code is available at https://github.com/XinbangZhang/DSO-NAS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, background): “searching for the neural architectures and succeeded to find the network for only one shot [7], [145].”
Federated Class-Incremental Learning (2022, methodology): “Moreover, [4] designs a layer-wise aggregation strategy to reduce computation overhead [55, 56].”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, methodology): “We note that sampling a path according to architecture distribution during optimization is already used in previous weight sharing approaches [19, 25, 26, 4, 29, 32].”
DATA: Differentiable ArchiTecture Approximation With Distribution Guided Sampling (2020, methodology): “21 7 gradient-based DSO-NAS-share [49] 25.”
AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction (2020, methodology): “Inspired by SNAS, we model the selection process of important feature interactions as a structural learning problem.”
ASAP: Architecture Search, Anneal and Prune (2019, methodology): “…2.55 ± 0.05 2.8 3150 NASNet-A (Zoph et al., 2018) 2.65 3.3 1800 PNAS (Liu et al., 2018a) 3.41 3.2 150 SNAS (Xie et al., 2019) 2.85 ± 0.02 2.8 1.5 DSO-NAS (Zhang et al., 2018) 2.95 ± 0.12 3 1 PARSEC (Casale et al., 2019) 2.81 ± 0.03 3.7 1 DARTS(2nd) (Liu et al., 2018b) 2.76 ± 0.06 3.4 1 PC-DARTS…”
Deeper Insights into Weight Sharing in Neural Architecture Search (2020, background): “t on candidate choices (Bender et al., 2018), converting the discrete search space into a differentiable one (Liu et al., 2018c; Wu et al., 2018; Xie et al., 2019), searching via sparse optimization (Zhang et al., 2018), and directly searching for child models for large-scale target tasks and hardwa”