GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet

By Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, Changshui Zhang. 2020

153 citations recorded by the source corpus.

Training a supernet matters for one-shot neural architecture search (NAS) methods since it serves as a basic performance estimator for different architectures (paths). Current methods mainly hold the assumption that a supernet should give a reasonable ranking over all paths. They thus treat all paths equally, and spare much effort to train paths. However, it is harsh for a single supernet to evaluate accurately on such a huge-scale search space (e.g., 7^21). In this paper, instead of covering all paths, we ease the burden of supernet by encouraging it to focus more on evaluation of those potentially-good ones, which are identified using a surrogate portion of validation data. Concretely, during training, we propose a multi-path sampling strategy with rejection, and greedily filter the weak paths. The training efficiency is thus boosted since the training space has been greedily shrunk from all paths to those potentially-good ones. Moreover, we further adopt an exploration and exploitation policy by introducing an empirical candidate path pool. Our proposed method GreedyNAS is easy-to-follow, and experimental results on ImageNet dataset indicate that it can achieve better Top-1 accuracy under same search space and FLOPs or latency level, but with only ~60% of supernet training cost. By searching on a larger space, our GreedyNAS can also obtain new state-of-the-art architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yunhe Wang
2. Fei Wang
3. Kai Han
4. Zhongzhan Huang
5. Chunjing Xu
6. Senwei Liang
7. Wei He
8. Jianyuan Guo

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Kai Han
3. Chunjing Xu
4. Jianyuan Guo
5. Dacheng Tao
6. Fei Wang
7. Chen Qian
8. Zhongzhan Huang

In Neural Architecture Search & AutoML

1. Yunhe Wang
2. Wei Zhang
3. Houwen Peng
4. Jianlong Fu
5. Tong Zhang

Citations in the corpus

• Distilling Object Detectors via Decoupled Features (2021, background): “Designing a detector coupled with lightweight backbone network [55, 41, 45, 32, 61, 64] is also a trend for faster detection speed.”
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020, background): “Neural architecture search [61, 27, 52, 54, 44] also provides more possibility of network architectures.”
• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, background): “With this aim, efforts have been made to design more compact networks, compress or prune a pretrained model [28, 29] or even search for a tiny backbone network [37, 39].”
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug (2020, methodology): “In fact, it is popular to use validation accuracy of a candidate network as a reward signal in NAS [24, 34, 10, 35, 32].”
• Local Correlation Consistency for Knowledge Distillation (2020, background): “Currently, many complicated neural networks with deeper and wider architectures have been proposed to pursuit high performance [25, 34].”
• Knowledge Distillation from A Stronger Teacher (2022, background): “The advent of automatic feature engineering fuels deep neural networks to achieve remarkable success in a plethora of computer vision tasks, such as image classification [17, 19, 38, 48, 53], object detection [2, 23], and semantic segmentation [5, 54].”
• LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “To mitigate memory consumption in scenarios with a vast selection array, techniques such as single-path sampling [15,57], binary approximation [1], and partial-channel usage [55] present viable solutions.”
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, methodology): “7 single path 10 12 GreedyNAS-A [28] 77.”