StacNAS: Towards stable and consistent optimization for differentiable Neural Architecture Search

By Guilin Li, Xing Zhang, Zitong Wang, Zhenguo Li, Tong Zhang. 2019

27 citations recorded by the source corpus.

Earlier methods for Neural Architecture Search were computationally expensive. Recently proposed Differentiable Neural Architecture Search algorithms such as DARTS can effectively speed up the computation. However, the current formulation relies on a relaxation of the original problem that leads to unstable and suboptimal solutions. We argue that these problems are caused by three fundamental reasons: (1) The difficulty of bi-level optimization; (2) Multicollinearity of correlated operations such as max pooling and average pooling; (3) The discrepancy between the optimization complexity of the search stage and the final training. In this paper, we propose a grouped variable pruning algorithm based on one-level optimization, which leads to a more stable and consistent optimization solution for differentiable NAS. Extensive experiments verify the superiority of the proposed method regarding both accuracy and stability. Our new approach obtains state-of-the-art accuracy on CIFAR-10, CIFAR-100 and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Zhenguo Li
3. Weinan Zhang
4. Wei Zhang
5. Yong Yu
6. Yunhe Wang
7. Dacheng Tao
8. Xiangxiang Chu

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Dacheng Tao
3. Zhenguo Li
4. Luc Van Gool
5. Yunhe Wang
6. Wei Zhang
7. Xiaodan Liang
8. Alan L. Yuille

Citations in the corpus

• NAS evaluation is frustratingly hard (2019, methodology): “The methods used are DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS and NAO.”
• Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (2020, background): “Neural Architecture Search [10, 11, 12, 13, 14] searches architectures that consume fewer resources with comparable performance.”
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction (2020, methodology): “Inspired by the recent work DARTS [9, 17, 20] for neural architecture search, we propose a two-stage method AutoFIS for automatic selecting low-order and high-order feature interactions in factorization models.”
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, background): “Some of them extend its use in higher-level architecture search spaces with performance awareness in mind [4,36], some learn a stochastic distribution instead of architectural parameters [36,37,40,11,12], and others offer remedies on discovering its lack of robustness [26,6,21,18,39].”