Single Path One-Shot Neural Architecture Search with Uniform Sampling
By Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei. 2019
1,036 citations recorded by the source corpus.
We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method, however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex search spaces (e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency). It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Dacheng Tao
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Trevor Darrell
- Jiashi Feng
- Alan L. Yuille
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Alan L. Yuille
- Trevor Darrell
- Chunhua Shen
- Yi Yang
In Neural Architecture Search & AutoML
Citations in the corpus
- FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “One category of NAS is differentiable neural architecture search (DNAS).”
- A Comprehensive Survey of Neural Architecture Search (2020, background): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
- GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “Instead of specifying an epoch number [11, 4], we use the proposed principle to stop the supernet training with tolerance α = 0.”
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “[1,3,9,22,25,30,35]) or otherwise post-process them (e.”
- Real-Time Federated Evolutionary Neural Architecture Search (2020, neutral): “To fix this issue, a sampling strategy is proposed by [39], [40], in which only one or two paths (a path is a sub-network connecting the inputs and outputs) are sampled from a complex neural network for training.”
- Semi-Supervised Neural Architecture Search (2020, background): “[18, 17, 15, 5, 36, 4, 27, 9] also leverage the one-shot idea to perform efficient search while using different search algorithms.”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Different from all differentiable NAS methods, one-shot NAS only do architecture optimization in one-shot, before which they obtain a rough estimation of the graph through either pretraining [1, 7] or an auxiliary hypernetwork [2].”
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, neutral): “(2019); Liu et al. (2019a)).”
- Auto-GNN: Neural architecture search of graph neural networks (2019, background): “Output statistics [25] is defined as the mean, variance, or interval of the output value in each graph convolutional layer of GNN architecture.”
- Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “Fine-tune weights and transformation matrix Shrink width: E = 6 E [6, 4] E [6, 4, 3] ⇒ ∈ ⇒ ∈”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “In practice, we set k = 6 to make it comparable to FairNAS. c) SPOS : an reimplementation of Single-Path One-Shot [15].”
- DetNAS: Backbone Search for Object Detection (2019, methodology): “In one-shot NAS methods [6, 2, 1], the search space is encoded in a supernet which consists of all candidate architectures.”
- Auto-Keras: An Efficient Neural Architecture Search System (2018, background): “evolutionary algorithms [12, 17, 30, 38, 39, 41], require a large n to reach a good performance.”
- Automated Machine Learning: From Principles to Practices (2018, methodology): “Therefore, a reasonable idea is to apply AutoML to search suitable PEFT methods [146, 171] and the parameters [91] that require updating for corresponding downstream tasks.”
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, methodology): “To reduce the NAS cost, supernet based methods (Pham et al., 2018; Bender et al., 2018; Guo et al., 2019) apply a proxy for sub-network performance and adopt search algorithms to find good sub-networks.”
- AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “Inspired from single-path NAS [19], AMER pre-allocates the weight matrix with maximum dimension of ( K 0 , K 0 ), i.e., W c ∈ R K 0 × K 0 for each hidden layer.”
- DetNAS: Neural Architecture Search on Object Detection (2019, methodology): “In DetNAS, we conduct the supernet training and network evaluation both in a path-wise manner [3], single path sampling.”
- AutoFormer: Searching Transformers for Visual Recognition (2021, methodology): “The implementation of evolution search follows the same protocol as in SPOS [16].”