Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation
By Xin Chen, Lingxi Xie, Jun Wu, Qi Tian. 2019
738 citations recorded by the source corpus.
Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Dacheng Tao
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Tie-Yan Liu
- Alan L. Yuille
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Alan L. Yuille
- Trevor Darrell
- Chunhua Shen
In Neural Architecture Search & AutoML
Citations in the corpus
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “In another example, the solution in P-DARTS [99] is to add search-space regular- ization, which executes operation-level dropout to control the proportion of skip-connect operations occurring during training and evaluation.”
- MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, background): “For example, task-agnostic NAS is proposed for the multi-task learning framework [6]; releasing the constraints of handdesigned heuristics [29] or alleviating the gap between the search accuracy and the evaluation accuracy are also promising directions [3, 14].”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “P-DARTS [43] starts with the deep gap between the search and evaluation of network architecture and improves DARTS.”
- Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “2, we represent a cell as a directed acyclic graph (DAG) [9], [11], [20], [21], where the nodes denote the feature maps in deep networks and the edges denote some computational operations, convolution.”
- Semi-Supervised Neural Architecture Search (2020, background): “[18, 17, 15, 5, 36, 4, 27, 9] also leverage the one-shot idea to perform efficient search while using different search algorithms.”
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “We use the open-source code MMDetection (Chen et al., 2019a).”
- DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “each selected architecture, we follow the configurations and hyper-parameters of the previous works [4, 20] for evaluation on different datasets.”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “1 GPU-days on CIFAR10, or 3 .”
- XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “For example [25, 49, 6, 3] used adam [20], and [29] used SGD with momentum.”
- Automated Machine Learning: From Principles to Practices (2018, methodology): “• Pre-training stage : the LLM is trained on a vast corpus of unlabeled text with self-supervised learning to learn language patterns and general knowledge.”
- Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, neutral): “Similar to the coupling problem of discrete architecture encoding and the sub-model weights [16, 33] in NAS (retraining or fine-tuning weights is needed), the activated channel weights and the discretized binary gates may not well accommodate to each other.”
- Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, background): “On the other hand, instead of searching over a discrete set of architecture candidates, differentiable methods [30, 38, 29] relax the search space to be continuous, such that the search can be optimized by the efficient gradient descent.”
- ScarletNAS: Bridging the Gap Between Scalability and Fairness in Neural Architecture Search (2019, background): “It is not desired that a model with too many identity layers excels in FLOPs while behaving poorly in accuracy, also pointed out in (Chen et al. 2019).”
- MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “NAS methods have produced highly competitive architectures on various computer vision tasks, such as image classification [5, 8, 9, 25, 35, 43, 59, 71], object detection [7, 14, 66], and semantic segmentation [33, 44].”
- StacNAS: Towards Stable and Consistent Differentiable Neural Architecture Search (2019, neutral): “8 1 PDARTS (base) Chen et al. (2019) 15.”
- Any-Precision Deep Neural Networks (2019, background): “We observe active researches [20, 2, 4] in looking for more effi-cient deep neural network architectures to support practical usage [15, 32, 26, 35].”