sharpDARTS: Faster and More Accurate Differentiable Architecture Search

By Andrew Hundt, Varun Jain, Gregory Hager. 2019

71 citations recorded by the source corpus.

Neural Architecture Search (NAS) has been a source of dramatic improvements in neural network design, with recent results meeting or exceeding the performance of hand-tuned architectures. However, our understanding of how to represent the search space for neural net architectures and how to search that space efficiently are both still in their infancy. We have performed an in-depth analysis to identify limitations in a widely used search space and a recent architecture search method, Differentiable Architecture Search (DARTS). These findings led us to introduce novel network blocks with a more general, balanced, and consistent design; a better-optimized Cosine Power Annealing learning rate schedule; and other improvements. Our resulting sharpDARTS search is 50% faster with a 20-30% relative improvement in final model error on CIFAR-10 when compared to DARTS. Our best single model run has 1.93% (1.98+/-0.07) validation error on CIFAR-10 and 5.5% error (5.8+/-0.3) on the recently released CIFAR-10.1 test set. To our knowledge, both are state of the art for models of similar size. This model also generalizes competitively to ImageNet at 25.1% top-1 (7.8% top-5) error. We found improvements for existing search spaces but does DARTS generalize to new domains? We propose Differentiable Hyperparameter Grid Search and the HyperCuboid search space, which are representations designed to leverage DARTS for more general parameter optimization. Here we find that DARTS fails to generalize when compared against a human's one shot choice of models. We look back to the DARTS and sharpDARTS search spaces to understand why, and an ablation study reveals an unusual generalization gap. We finally propose Max-W regularization to solve this problem, which proves significantly better than the handmade design. Code will be made available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “[127] propose SharpDARTS, which uses a general, balanced and consistent design, namely SharpConv Block.”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “Moreover, recent proposed NAS methods achieve a higher accuracy than our method [20, 11, 2].”
XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “[17, 25], is not suitable for the unique structure of the architecture space.”
ASAP: Architecture Search, Anneal and Prune (2019, methodology): “cosine annealing learning rate [18], with amplitude decay factor of 0.”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, background): “…realm of training methodologies, prior research has delved into diverse learning rate schedulers (LRS) (Howard & Ruder, 2018; Raffel et al., 2020; Hundt et al., 2019), with the Cosine LRS (Loshchilov & Hutter, 2016) emerging as the predominant choice in LLMs. Kaplan et al. (2020) and Hoffmann et…”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, methodology): “While concurrent methods consider relaxed convolution formulations with insight similar to our work [33], [34], [35], [21], they either use design spaces and objectives that have been shown to be hardware inefﬁcient ( e.g. , cell-based space, FLOP count), or they optimize over a subset of our…”