SNAS: Stochastic Neural Architecture Search

By Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin. International Conference on Learning Representations, 2018

1,014 citations recorded by the source corpus.

We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline. In this work, NAS is reformulated as an optimization problem on parameters of a joint distribution for the search space in a cell. To leverage the gradient information in generic differentiable loss for architecture search, a novel search gradient is proposed. We prove that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable reward to enforce a resource-efficient constraint. In experiments on CIFAR-10, SNAS takes less epochs to find a cell architecture with state-of-the-art accuracy than non-differentiable evolution-based and reinforcement-learning-based NAS, which is also transferable to ImageNet. It is also shown that child networks of SNAS can maintain the validation accuracy in searching, with which attention-based NAS requires parameter retraining to compete, exhibiting potentials to stride towards efficient NAS on big datasets. We have released our implementation at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

View Paper: https://arxiv.org/abs/1812.09926

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Dacheng Tao
  3. Mohit Bansal
  4. Yoshua Bengio
  5. Jianfeng Gao
  6. Percy Liang
  7. Sergey Levine
  8. Trevor Darrell

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Yi Yang
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Trevor Darrell
  7. Alan L. Yuille
  8. Jitendra Malik

In Neural Architecture Search & AutoML

  1. Frank Hutter
  2. Colin White
  3. Yash Savani
  4. Xiaojun Chang
  5. Quoc V. Le
  6. Xuanyi Dong
  7. Martin Wistuba
  8. Arber Zela

Citations in the corpus