Searching for a Robust Neural Architecture in Four GPU Hours

By Xuanyi Dong, Yezhou Yang. 2019

719 citations recorded by the source corpus.

Conventional neural architecture search (NAS) approaches are usually based on reinforcement learning or evolutionary strategy, which take more than 1000 GPU hours to find a good model on CIFAR-10. We propose an efficient NAS approach, which learns the searching approach by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG). This DAG contains thousands of sub-graphs, each of which indicates a kind of neural architecture. To avoid traversing all the possibilities of the sub-graphs, we develop a differentiable sampler over the DAG. This sampler is learnable and optimized by the validation loss after training the sampled architecture. In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS). In experiments, we can finish one searching procedure in four GPU hours on CIFAR-10, and the discovered model obtains a test error of 2.82% with only 2.5M parameters, which is on par with the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Yi Yang
  3. Jiashi Feng
  4. Dacheng Tao
  5. Xiaodan Liang
  6. Trevor Darrell
  7. Mohit Bansal
  8. Jianfeng Gao

In Computer Vision and Pattern Recognition

  1. Yi Yang
  2. Luc Van Gool
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Trevor Darrell
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Neural Architecture Search & AutoML

  1. Xiaojun Chang
  2. Colin White
  3. Yash Savani
  4. Frank Hutter
  5. Xuanyi Dong
  6. Arber Zela
  7. Bo Zhang
  8. Xiangxiang Chu

Citations in the corpus