One-Shot Neural Architecture Search via Self-Evaluated Template Network

By Xuanyi Dong, Yezhou Yang. 2019

208 citations recorded by the source corpus.

Neural architecture search (NAS) aims to automate the search procedure of architecture instead of manual design. Even if recent NAS approaches finish the search within days, lengthy training is still required for a specific architecture candidate to get the parameters for its accurate evaluation. Recently one-shot NAS methods are proposed to largely squeeze the tedious training process by sharing parameters across candidates. In this way, the parameters for each candidate can be directly extracted from the shared parameters instead of training them from scratch. However, they have no sense of which candidate will perform better until evaluation so that the candidates to evaluate are randomly sampled and the top-1 candidate is considered the best. In this paper, we propose a Self-Evaluated Template Network (SETN) to improve the quality of the architecture candidates for evaluation so that it is more likely to cover competitive candidates. SETN consists of two components: (1) an evaluator, which learns to indicate the probability of each individual architecture being likely to have a lower validation loss. The candidates for evaluation can thus be selectively sampled according to this evaluator. (2) a template network, which shares parameters among all candidates to amortize the training cost of generated candidates. In experiments, the architecture found by SETN achieves the state-of-the-art performance on CIFAR and ImageNet benchmarks within comparable computation costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “SETN [91] noted that it is extremely difficult to find the best architecture from these sampled candidate architectures.”
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “For each architecture candidate, we can calculate its probability of being a good architecture from the architecture encoding following SETN (Dong & Yang, 2019a).”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “5 Single path Gradient SETN (T=1K) [15] 2.”
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, background): “Since then, many subsequent works have been dedicated to accelerating the process (Dong & Yang, 2019b), reducing memory cost (Xu et al., 2020), or fostering its ability such as hardware-awareness (Cai et al., 2019; Wu et al., 2019), ﬁner granularity (Mei et al., 2020) and so on.”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “By enabling the gradient descent for the architecture optimization, DARTS signiﬁcantly reduces the search cost to several GPU hours (Liu et al., 2019; Xu et al., 2020; Dong & Yang, 2019a).”
Neural Architecture Search without Training (2020, unknown): “(Dong & Yang, 2019a), and ENAS (Pham et al., 2018).”
AdversarialNAS: Adversarial Neural Architecture Search for GANs (2019, background): “For NAS in the classification task [25, 7, 4], the inner optimization (Eq.”
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, methodology): “SETN [7] combines one-shot methodology [3, 1] with DARTS [18] by learning a stochastic evaluator to sample models of low validation loss.”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, background): “(IV) Differentiable algorithms. e.g., first order DARTS (DARTS-V1) [8], second order DARTS (DARTS-V2), GDAS [7], SETN [17], TAS [21], FBNet-V2 [44], TuNAS [49].”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “NAS methods have produced highly competitive architectures on various computer vision tasks, such as image classification [5, 8, 9, 25, 35, 43, 59, 71], object detection [7, 14, 66], and semantic segmentation [33, 44].”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, background): “Recent works propose the one-shot approaches [1,4,8,24,27,35] in NAS to reduce the computation cost.”