DARTS: Differentiable Architecture Search

By Hanxiao Liu, Karen Simonyan, Yiming Yang. 2018

5,020 citations recorded by the source corpus.

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

Subfields: Computer Vision and Pattern Recognition, Immunology, Molecular Biology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Immunology

In Molecular Biology

In Model Architectures

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained architectural principles at a larger scale (e.g., Liu et al., 2018; Real et al., 2018; Zoph et al., 2018).”
AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “To accelerate NAS, one-shot models [29, 26, 2], DARTS [22] and NAS with Transferable Cell [40, 20] were proposed.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “In other words: Then, again following [30], we approximate this bilevel optimization by a single virtual training step, where ξ is the virtual learning rate.”
True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
Dynamic Neural Networks: A Survey (2021, background): “The recent research on neural architecture search (NAS) [9], [10] further speeds up the process of designing more powerful structures.”
Designing Network Design Spaces (2020, methodology): “We observe that R EG N ETS are surprisingly effective in this regime considering the substantial body of work on ﬁnding better mobile networks via both manual design [9, 25, 19] and NAS [35, 23, 17, 18].”
On Network Design Spaces for Visual Recognition (2019, methodology): “As a case study of our methodology, we examine the network design spaces used by several recent methods for neural architecture search (NAS) [39, 29, 20, 28, 21].”
Squeeze-and-Excitation Networks (2017, methodology): “With the goal of reducing the computational burden of these methods, efficient alternatives to this approach have been proposed based on Lamarckian inheritance [32] and differentiable architecture search [33].”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “These approaches are mainly based on reinforcement learning [35, 36, 1, 19, 25], evolutionary search [26], differentiable search [21], or other learning algorithms [19, 17, 23].”
Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “The widely used methods contains reinforcement learning (RL) [5], [6], [8], [9], [16], evolutionary algorithm (EA) [17]–[23], and gradient descent (GD) [10], [24], [25], Bayesian optimization [26]– [32] and so on.”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “1 588 225 SMBO complete DARTS (Liu et al., 2019) 26.”
FedNAS: Federated Deep Learning via Neural Architecture Search (2020, methodology): “Our search space follows the mixed-operation search space defined in DARTS [8] and MiLeNAS [3], where we search in two shared convolutional cells and then build it up as an entire model architecture (as shown in Figure 2).”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “∗Equal contribution Formally, gradient-based methods can be formulated as a bilevel optimization problem [17]:”
Weight Agnostic Neural Networks (2019, methodology): “While our search method is based on evolution, future work may extend the approach by incorporating recent ideas that formulate architecture search in a differentiable manner [65] to make the search more efﬁcient.”
Neural Architecture Optimization (2018, methodology): “One one hand, both NAO and DARTS conducts continuous optimization via gradient based method; on the other hand, the continuous space in the two works are different: in DARTS it is the mixture weights and in NAO it is the embedding of neural architectures.”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture (Stanley & Miikkulainen, 2002; Liu et al., 2018a; Real et al., 2018; Miikkulainen et al., 2017; Xie & Yuille, 2017).”