Understanding and Robustifying Differentiable Architecture Search
By Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter. 2019
415 citations recorded by the source corpus.
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Chunhua Shen
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Andrea Vedaldi
- Alan L. Yuille
In Safety, Robustness & Interpretability
In Neural Architecture Search & AutoML
- Frank Hutter
- Marius Thomas Lindauer
- Matthias Feurer
- Arber Zela
- Xia Hu
- Qingquan Song
- Katharina Eggensperger
- Xiaojun Chang
Citations in the corpus
- A Comprehensive Survey of Neural Architecture Search (2020, positive): “e normalization to solve this problem. It reduces the uncertainty in the search process by adding a set of edge-level parameters. Therefore, PC-DARTS can save memory and is more efficient and stable. [110] recent found that DARTS [17] has a poor test performance for the architecture generated in a w”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “For instance, architectures from DARTS usually contain an excessive number of skip connections [48,23], which damage the outcome performance.”
- Automated Machine Learning: From Principles to Practices (2018, background): “Sparse coding [35] is a representation learning method that aims at finding a sparse representation of the input data in the form of a linear combination of basic elements.”
- Differentiable Feature Aggregation Search for Knowledge Distillation (2020, methodology): “Among the one-shot methods, the differentiable architecture search (DARTS) [5,22,24,26,36,40] further relaxes the discrete search space to be continuous and couples the architecture parameters with the model parameters in the supernet.”
- Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “A recent focus in NAS research has been on computational efficiency with algorithms utilizing gradient-based optimization [8, 12, 36, 55, 59, 56], network morphisms [7, 13, 14], or multi-fidelity optimization [3, 17, 60].”
- SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search (2019, background): “To this end, skip connections are commonly used in differentiable approaches [2, 33], but they face a common issue of undesired skip connection aggregation as noted by [3, 36], which yields nonoptimal results.”
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (2020, methodology): “…the correlation between the lower fidelity proxy model used in architecture search and the final model from architecture evaluation in the Appendix H. Zela et al. (2020) observed that the generalization performance of architectures found by DARTS heavily depends on the hyperparameters used during…”
- When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “It has been extensively studied that the convergence to a flat region whose curvature is small benefits the generalization of neural networks (Keskar et al., 2017; Kleinberg et al., 2018; Jastrz˛ebski et al., 2019; Chen & Hsieh, 2020; Smith & Le, 2018; Zela et al., 2020; Chaudhari et al., 2017).”
- iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “Different from the majority of existing works that attributes the failure of DARTS to its supernet optimization (Zela et al., 2020a; Benyahia et al., 2019), or the final discretization with argmax (Chen & Hsieh, 2020; Wang et al., 2021b), this paper revisits DARTS from the perspective of the…”
- Hierarchical Neural Architecture Search for Deep Stereo Matching (2020, result): “Similar observations, albeit in another context, are reported in [34].”
- Neural Architecture Search with GBDT (2020, background): “Search Space Search Search space plays an essential role in the search phase [34, 31].”
- Propagation Model Search for Graph Neural Networks (2020, result): “This observation is consistent with (Zela et al. 2020) and supports the motivation of our search algorithm.”
- SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, background): “…et al., 2018), Dragonfly (Kandasamy et al., 2020) and BOHB (Falkner et al., 2018) on the surrogate benchmark for HPO on DNNs on the letter dataset (Falkner et al., 2018), a joint HPO+NAS benchmark (Klein and Hutter, 2019) on the Naval-Propulsion dataset and a pure NAS benchmark (Zela et al., 2020).”
- Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, methodology): “To avoid these two major issues, we propose a novel approach named Fair DARTS, in which we discuss current robustifying methods [5, 16, 32] on DARTS with a unified fairness perspective.”