DARTS+: Improved Differentiable Architecture Search with Early Stopping

By Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, Zhenguo Li. 2019

317 citations recorded by the source corpus.

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

NAS evaluation is frustratingly hard (2019, background): “Research in the last two years has proceeded at a rapid pace and many search strategies have been proposed, from Rein-forcement Learning (Zoph & Le, 2017; Pham et al., 2018), to Evolutionary Algorithms (Real et al., 2017), to Gradient-based methods (Liu et al., 2019; Liang et al., 2019).”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “The similar idea of optimizing optimal subgraphs within a supergraph is also adopted by Liu et al. (2019a); Wu et al. (2019); Guo et al.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “[32] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li.”
Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, positive): “The very recent method of Darts+ [16], which is concurrent to this work here, presents similar observations.”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “For instance, architectures from DARTS usually contain an excessive number of skip connections [52, 26], which damage the outcome performance.”
Automated Machine Learning: From Principles to Practices (2018, methodology): “Supernet training is another parameter reusing technique that is usually applied in neural architecture search.”
Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement (2020, methodology): “By adopting the continuous relaxation technique used in differentiable NAS literature [16, 39, 15], we introduce the vectorized form α = {αt,αn} to encode architectures in our search space (denoted as A) for CIEM and CNRM, respectively.”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, background): “(Zela et al., 2020; Liang et al., 2019) reduced the number of skip-connect operations by early stopping.”
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, background): “2 Performance Collapse Caused by Intractable Skip Connections DARTS suffers from significant performance decay when skip connections become dominant [4, 18].”
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, methodology): “Thirdly, extra computing cost (Zela et al., 2020) or careful tuning of hyper-parameters (Chen et al., 2019b; Liang et al., 2019) are required.”
ASAP: Architecture Search, Anneal and Prune (2019, unknown): “09 4.0 0.82 DARTS+ (Liang et al., 2019) 2 .”
StacNAS: Towards Stable and Consistent Differentiable Neural Architecture Search (2019, background): “(2019); Liang et al. (2019)) and in some cases even no better than random search (Li & Talwalkar (2019)).”
Rethinking Graph Neural Architecture Search from Message-passing (2021, background): “This is discussed in work [19]: after certain searching epochs, the number of identity operation increases dramatically in the selected architecture, which results in poor performance of the selected architecture.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “Approximated proxy inference such as truncated training/early stopping can accelerate the search, but is well known to introduce search bias to the inaccurate results obtained (Pham et al., 2018; Liang et al., 2019; Tan et al., 2020).”