Understanding and Robustifying Differentiable Architecture Search

By Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter. 2019

415 citations recorded by the source corpus.

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Tie-Yan Liu
  2. Quoc V. Le
  3. Tao Qin
  4. Yoshua Bengio
  5. Frank Hutter
  6. Qiang Yang
  7. Dacheng Tao
  8. Mohit Bansal

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Chunhua Shen
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Trevor Darrell
  7. Andrea Vedaldi
  8. Alan L. Yuille

In Safety, Robustness & Interpretability

  1. Tong Zhang
  2. Frank Hutter
  3. Boqing Gong
  4. Xiaojun Chang
  5. Jian Tang
  6. Qiang Yang
  7. Yee Whye Teh
  8. Cho-Jui Hsieh

In Neural Architecture Search & AutoML

  1. Frank Hutter
  2. Marius Thomas Lindauer
  3. Matthias Feurer
  4. Arber Zela
  5. Xia Hu
  6. Qingquan Song
  7. Katharina Eggensperger
  8. Xiaojun Chang

Citations in the corpus