ASAP: Architecture Search, Anneal and Prune

By Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, Lihi Zelnik-Manor. 2019

108 citations recorded by the source corpus.

Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with $0.2$ GPU search days we achieve an error rate of $1.68\%$ on CIFAR-10.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quanming Yao
2. Quoc V. Le
3. Junjie Yan
4. Chen Liang
5. David R. So
6. Wei Wu
7. Tal Ridnik
8. Asaf Noy

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Dacheng Tao
4. Wanli Ouyang
5. Bernt Schiele
6. Ming-Hsuan Yang
7. Yi Yang
8. Chunhua Shen

Citations in the corpus

• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Orthogonally, a few studies benefit from extending the search space to two such aspects simultaneously (Zela et al., 2018; Miikkulainen et al., 2019; Noy et al., 2019).”
• DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, neutral): “3 gradient ASAP [25] CIFAR10 2.”
• XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “a pooling layer), as the first requires some period for training, as stated by [29, 17].”
• Differentiable Neural Architecture Search via Proximal Iterations (2019, background): “Due to the differentiable relaxation in (1), an ensemble of operations are maintained and all operations in the search space need to be forward and backward-propagated when updating w; the evaluation of the second order term in (3) is very expensive known as a computation bottleneck of DARTS [39, 29”
• Improving One-Shot NAS by Suppressing the Posterior Fading (2019, methodology): “Other relevant works are ASAP [26] and XNAS [25], which introduce pruning during the training of over-parameterized networks to improve the efficiency of NAS.”
• Graph-Guided Architecture Search for Real-Time Semantic Segmentation (2019, background): “What’s more, there are also some excellent works [10, 32] to reduce the difficulty of optimization by decreasing gradually the size of search space.”
• Efficient Neural Architecture Search via Proximal Iterations (2019, background): “Architecture Test Error (%) Para (M) Ops Nodes Time (GPU days) ASAP (Noy et al. 2019) 3.”