Zero-Cost Proxies for Lightweight NAS
By Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, Nicholas Donald Lane. International Conference on Learning Representations, 2021
352 citations recorded by the source corpus.
Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4× quicker than the best previous result.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/6dc84d38f6d8d964456b127d6f45c5b4de73bb86
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Wuyang Chen
- Zhangyang Wang
- Denny Zhou
- Elliot J. Crowley
- Jack Turner
- Xiaodan Song
- Amos J. Storkey
- Wei Huang
In Computer Vision and Pattern Recognition
- Zhangyang Wang
- Wuyang Chen
- Xianzhi Du
- Elliot J. Crowley
- Jack Turner
- Xiaodan Song
- Amos J. Storkey
- Wei Huang
Citations in the corpus
- Neural Architecture Search without Training (2020, methodology): “For comparison, we include the best-performing architecture scoring functions from Abdelfattah et al. (2021) — grad norm and synflow — as baselines.”
- Auto-scaling Vision Transformers without Training (2022, methodology): “We note a recent surge of training-free neural architecture search methods for ReLU-based CNNs, leveraging local linear maps (Mellor et al., 2020), gradient sensitivity (Abdelfattah et al., 2021), number of linear regions (Chen et al., 2021e;f), or network topology (Bhardwaj et al., 2021).”
- NeurIDA: Dynamic Modeling for Effective In-Database Analytics (2025, unknown): “It then applies Zero-Cost Proxies (ZCP) [5, 42], techniques from Neural Architecture Search [43, 52] that estimate model performance without full training, to quickly score each candidate base model, yielding a proxy score 𝑠 𝑖 .”
- pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “…vision tasks, zero-cost proxies are often used in relatively static roles (e.g., to pre-train or warm-start search strategies) (Mellor et al., 2021; Abdelfattah et al., 2021), when directly transferred to tabular NAS, their inherent estimation noise makes such static usage insufficient to deliver…”