PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
By Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, H. Xiong. 2019
703 citations recorded by the source corpus.
Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture. In this paper, we present a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We solve it by introducing edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability. Experiment results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Xiaodan Liang
- Quoc V. Le
- Qiang Yang
- Frank Hutter
- Yoshua Bengio
- Xiaojun Chang
- Colin White
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Xiaodan Liang
- Yi Yang
- Alan L. Yuille
- Zhangyang Wang
- Ming-Hsuan Yang
- Chunhua Shen
In Efficiency & Compression
In Neural Architecture Search & AutoML
- Colin White
- Yash Savani
- Arber Zela
- Frank Hutter
- Xiaojun Chang
- Julien N. Siems
- Willie Neiswanger
- Quanming Yao
Citations in the corpus
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In order to reduce DARTS’s memory usage during search and improve search efficiency, unlike GDAS sampling subgraphs in DAG and training only one subgraph in one iteration, PC-DARTS [83] chooses to start from the channel.”
- Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “2, we represent a cell as a directed acyclic graph (DAG) [9], [11], [20], [21], where the nodes denote the feature maps in deep networks and the edges denote some computational operations, e.”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “2020) that BANANAS was competitive with nine other popular NAS algorithms, including one-shot methods (Liu, Simonyan, and Yang 2018; Xu et al. 2019; Dong and Yang 2019).”
- Automated Machine Learning: From Principles to Practices (2018, background): “Sparse coding [35] is a representation learning method that aims at finding a sparse representation of the input data in the form of a linear combination of basic elements.”
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (2020, methodology): “For instance, while PC-DARTS performs the best in search space 1, this is clearly not the case in the other search spaces.”
- LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “To mitigate memory consumption in scenarios with a vast selection array, techniques such as single-path sampling [15,57], binary approximation [1], and partial-channel usage [55] present viable solutions.”
- NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization (2021, methodology): “For the methodology, the proposed ordered dropout is most similar to the partial channel connections [24].”
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, methodology): “05 training-free PC-DARTS (ImageNet) (Xu et al., 2019)† 24.”
- DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, methodology): “(f) Reduction Figure 14: Found normal cells and reduction cells by PC-DARTS (Xu et al., 2020) without channel shuffling in the DARTS’ standard search space on CIFAR-10 dataset.”
- Rethinking Graph Neural Architecture Search from Message-passing (2021, background): “The search strategy details how to explore the search space, which is mainly classified into reinforcement learning (RL) [3, 43, 44], evolutionary algorithms (EA) [20, 29, 30] and gradient-based (GB) [15, 21, 37, 39] methods.”