MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

By Chaoyang He, Haishan Ye, Li Shen, Tong Zhang. 2020

146 citations recorded by the source corpus.

Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of second-order methods. In this paper, we demonstrate that gradient errors caused by such approximations lead to suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. To remedy this, this paper proposes MiLeNAS, a mixed-level reformulation for NAS that can be optimized efficiently and reliably. It is shown that even when using a simple first-order method on the mixed-level formulation, MiLeNAS can achieve a lower validation error for NAS problems. Consequently, architectures obtained by our method achieve consistently higher accuracies than those obtained from bilevel optimization. Moreover, MiLeNAS proposes a framework beyond DARTS. It is upgraded via model size-based search and early stopping strategies to complete the search process in around 5 hours. Extensive experiments within the convolutional architecture search space validate the effectiveness of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “FedNAS [5] is a federated neural architecture search algorithm [130] that can help scattered workers collaboratively searching for a better architecture with higher accuracy in the non-I.”
FedNAS: Federated Deep Learning via Neural Architecture Search (2020, methodology): “As the process shown in Figure 1, in FedNAS, we first utilize the gradient-based method MiLeNAS [3] as local searcher on the local data of each worker.”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, methodology): “As seen in both manually designed neural architectures (ResNet [1]) and automated architectures discovered by neural architecture search (DARTS [2], MiLeNAS [3], EfficientNets [4]), scaling up CNN size (e.”
Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search. (2020, methodology): “As the process shown in Figure 1, in FedNAS, we first utilize the gradient-based method MiLeNAS [4] as local searcher on the local data of each worker.”
AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction (2020, methodology): “r model performance, as we face an incredibly huge (2C m 2 , with m the number of feature fields, if we only consider2nd-order feature interactions) space to search. Inspired by the recent work DARTS [9, 17, 20] for neural architecture search, we propose a two-stage method AutoFIS for automatic sele”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “This challenge can be further exacerbated when applying DNAS to DRL, as effective DNAS requires unbiased and stable (with low variance) gradient estimation [5, 26] to guarantee the convergence while such a requirement is often not feasible in a DRL environment, especially in real-world applications.”
Self-supervised Cross-silo Federated Neural Architecture Search (2021, methodology): “Other parameter settings are introduced from the original paper of MiLeNAS [50] and DARTS [59].”
SGAS: Sequential Greedy Architecture Search (2019, background): “In Differentiable Architecture Search (DARTS) [33] and its variants [55, 9, 58, 17], the optimal architecture is derived from a discrete search space by relaxing the selection of operations to a continuous optimization problem.”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “As seen in both manually designed neural architectures (ResNet [1]) and automated architectures discovered by neural architecture search (DARTS [2], MiLeNAS [3], EfficientNets [4]), scaling up CNN size (e.”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, background): “Recently, many neural architecture search (NAS) methods have emerged to jointly learn the weights and the network architectures [4, 6, 12, 14, 22, 31, 34, 41, 44, 45, 71].”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “To reduce the search cost, recent NAS methods employ dedicated networks to learn continuous representations of neural architectures and perform architecture search in continuous search spaces [14, 15, 16, 17].”
CARS: Continuous Evolution for Efficient Neural Architecture Search (2019, methodology): “To accelerate this process and make it automated, network architecture search (NAS) [44, 14, 36, 20, 16] has been proposed, and the learned architectures have exceeded those humandesigned architectures on a variety of tasks.”
From distributed machine learning to federated learning: a survey (2021, methodology): “During the training process, the parameters or the model structure of the machine learning model [61,68] is adjusted based on a training algorithm in order to improve the performance, e.”